Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

the: Jul 18, 2004; by Cowcaster

To append onto the same issue, I'm trying a different approach. The query returns only 500 entries at a time, even if there's more than that, so I have to loop through until I reach the end. Here's the entire code so you can understand what the terms are. In this instance, total = somewhere in the 37,xxx range, so testdict at the end should be around that size, yet it's staying at 8 entries. What's going wrong?

code:

import beatbox

sf_username = "username
sf_password = "password"
sf_api_token = "token"

sf = beatbox._tPartnerNS
svc = beatbox.Client()
svc.login("username","password")

query = svc.query("SELECT AccountId, CreatedById, CreatedDate, Field, NewValue, OldValue 
FROM AccountHistory WHERE Account.Industry = \'Government\' AND Account.Type = \'Prospect\'")

testdict = query['records']
total = query['size']
queryL = query['queryLocator']

while query['done'] is False and len(testdict) < int(str(total)):
	query = self._service.queryMore(queryL)
	queryL = query['queryLocator']
	testdict = testdict + query['records']

edit: I'm trying to use this example: http://tomhayden3.com/2013/08/04/salesforce-python/

---------
edit: I found out that it wasn't even entering the while loop, so I removed "query['done']" from the conditions. However, now it is popping errors at the line "query = self._service.queryMore(queryL)", saying that "NameError: name 'self' is not defined."

the fucked around with this message at 20:57 on Jun 13, 2014

# ? Jun 13, 2014 20:30

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 05:12

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

KICK BAMA KICK posted:

The interpreter always provides the object itself as the first argument to the method -- no matter how you invoke the method it can't be omitted as long as the method isn't defined as static, so the default value of None is never used.

code:

Python 3.3.1 (v3.3.1:d9893d13c628, Apr  6 2013, 20:25:12) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class c:
...     def __init__ (self = None):
...         self.x = 0
...
>>> c.__init__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __init__
AttributeError: 'NoneType' object has no attribute 'x'
>>>

# ? Jun 13, 2014 21:10

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

the posted:

edit: I found out that it wasn't even entering the while loop, so I removed "query['done']" from the conditions. However, now it is popping errors at the line "query = self._service.queryMore(queryL)", saying that "NameError: name 'self' is not defined."

Well, it isn't defined, so this is not very surprising. There's no "self" variable defined on any of the lines above that one. I don't know what the guy on that web page was thinking, but his code seems wrong at a cursory glance.

Given that you want to use the queryMore() method though, and given that based on a quick look at Beatbox's Github query() and queryMore() are both methods of the same class, and given that there is a variable called "service" in his code, I wonder whether just changing "self._service" to "service" would work. Or in your case "svc".

# ? Jun 13, 2014 21:22

coaxmetal: Oct 21, 2010; I flamed me own dad

QuarkJets posted:

List comprehensions are my favorite Python feature

e: And as an addition to the poster above me, short variable names can make it impossible to modify your code later. If you have to search for a variable called 'd' then you're pretty much hosed if you have any variables with a d in their name

Yeah, using descriptive names is something that I really like. I really like readable code. There is a good talk about python naming and readability from pycon a couple years ago if you want to watch a long video about it
https://www.youtube.com/watch?v=YklKUuDpX5c

I have coworkers who write with lovely style, and I have to get mad at them and fix it. Stuff like, putting an entire multiple element dictionary on a single line, or multiple k/v pairs per line.

# ? Jun 13, 2014 21:49

the: Jul 18, 2004; by Cowcaster

Hammerite posted:

Given that you want to use the queryMore() method though, and given that based on a quick look at Beatbox's Github query() and queryMore() are both methods of the same class, and given that there is a variable called "service" in his code, I wonder whether just changing "self._service" to "service" would work. Or in your case "svc".

I changed the line to:

code:

query = svc.queryMore(queryL)

And it pops this error:

TypeError: coercing to Unicode: need string or buffer, instance found

So I changed it to:

code:

query = svc.queryMore(str(queryL))

And now I'm popping an error on: testdict = testdict + query['records']

"TypeError: unsupported operand type(s) for +: 'instance' and 'instance'"

I guess I can't add those two together?

# ? Jun 14, 2014 00:59

Dominoes: Sep 20, 2007

Looking for advice on asynchronous behavior. I recently implemented a streaming stock data algorithm in a program. I created a generator function that yields data from a Requests stream. I created a thread (Qthread) that runs this generator in a for loop, and reports data to my main thread and GUI using QT signals.

My broker recommends using eventing, ie Java's Jetty, Ruby's EventMachine. I don't have a good grasp on events (or threads), but I do understand that the QT GUI uses events, called an event loop, that allows multitasking without threads. I feel like threads are more direct to the hardware and change settings with how the CPU runs things, while eventing might be simpler and more appropriate. Is there a good way to handle this in Python using events, or is my current approach using a thread appropriate?

# ? Jun 14, 2014 02:38

namaste friends: Sep 18, 2004; by Smythe

Can you use 'with'?

http://preshing.com/20110920/the-python-with-statement-by-example/

# ? Jun 14, 2014 03:14

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

the posted:

I changed the line to:
code:
query = svc.queryMore(queryL)
And it pops this error:

TypeError: coercing to Unicode: need string or buffer, instance found

So I changed it to:
code:
query = svc.queryMore(str(queryL))
And now I'm popping an error on: testdict = testdict + query['records']

"TypeError: unsupported operand type(s) for +: 'instance' and 'instance'"

I guess I can't add those two together?

I can't help you very much here. It is strange to me that the things you are adding are "instances", rather than dicts, since according to the variable names they are expected to be dicts. If they were dicts then you could do testdict.update(query['records']), but since they apparently are not dicts I do not know that this will work.

# ? Jun 14, 2014 14:36

Dominoes: Sep 20, 2007

Cultural Imperial posted:

Can you use 'with'?

http://preshing.com/20110920/the-python-with-statement-by-example/

How would that work? Wrap the main run loop in a 'with' generator?

# ? Jun 15, 2014 12:37

Crosscontaminant: Jan 18, 2007

Hammerite posted:

code:

Python 3.3.1 (v3.3.1:d9893d13c628, Apr  6 2013, 20:25:12) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class c:
...     def __init__ (self = None):
...         self.x = 0
...
>>> c.__init__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __init__
AttributeError: 'NoneType' object has no attribute 'x'
>>>

Why would you ever execute a dunder method directly?

code:

Python 3.4.1 (default, May 19 2014, 17:23:49) 
[GCC 4.9.0 20140507 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class c:
...     def __init__(self=None):
...         self.x = 0
... 
>>> c()
<__main__.c object at 0x7fcc026f7e80>
>>> c().x
0

Since __init__ always gets passed the object created by __new__ as the first parameter, self=None is just dummy code.

# ? Jun 15, 2014 14:22

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

Crosscontaminant posted:

Why would you ever execute a dunder method directly?

You wouldn't. Did you read the post I was replying to?

# ? Jun 15, 2014 15:25

the: Jul 18, 2004; by Cowcaster

Hammerite posted:

I can't help you very much here. It is strange to me that the things you are adding are "instances", rather than dicts, since according to the variable names they are expected to be dicts. If they were dicts then you could do testdict.update(query['records']), but since they apparently are not dicts I do not know that this will work.

Hmm, well, thanks anyway.

What is this guy doing differently, then? Because I'm pretty much using his code, and according to him it should add with no problem. Am I missing something?

# ? Jun 17, 2014 16:30

the: Jul 18, 2004; by Cowcaster

SurgicalOntologist, in the last page you suggested I try this in order to connect the list items:

code:

all_records = [', '.join(rec) for rec in qr1[sf.records:]]

I tried that, shown in the sequence below:

code:

while len(testdict) < int(str(total)):
	query = svc.queryMore(str(queryL))
	queryL = query['queryLocator']
	all_records = [', '.join(str(rec)) for rec in query[sf.records:]]
	list1.append(all_records)

The problem is that individual list entries are now coming out like this:

code:

In [16]: list1[1][1]
Out[16]: 'A, c, c, o, u, n, t, H, i, s, t, o, r, y, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 6, 
u, U, B, q, A, A, M, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 6, u, 7, g, P, A, A, Q, 2, 0, 1, 2,
 -, 0, 3, -, 2, 7, T, 1, 8, :, 2, 7, :, 0, 6, ., 0, 0, 0, Z, c, r, e, a, t, e, d'

# ? Jun 17, 2014 17:07

SurgicalOntologist: Jun 17, 2004

Looks like each rec is a string--your original post made it look like each rec was a list of strings.

is query[sf.records:] the list of strings?

In that case it would be

Python code:

all_records = ', '.join(query[sf.records:])

In any case you should try to understand what data type you have at each step in your procedure. Looping over strings gets you individual characters, so doing that inside ', '.join will put a comma in-between every character. On the other hand, looping over a list of strings (what the join does internally) will put a comma between every string in your list.

I also have to ask... why are you doing int(str(total))? I can't think of any conceivable situation in which you'd want to convert something to a string and then to an integer.

SurgicalOntologist fucked around with this message at 17:45 on Jun 17, 2014

# ? Jun 17, 2014 17:43

the: Jul 18, 2004; by Cowcaster

Thanks, I think my big problem is, like you said, wondering what kind of data I'm dealing with. Everything is being returned as instances, so I have to put a str() before everything.

So if I just print out query[sf.records:]:

code:

In [31]: query[sf.records:][0]
Out[31]: <records xsi:type="sf:sObject">...</records>

But if I make it a string:

code:

In [32]: str(query[sf.records:][0])
Out[32]: 'AccountHistory001000000072DKCAA200500000006wKQqAAM2013-10-03T12:39:52.000ZBillingCityNicoma Park'

Which is still not what I want, because they're all mashed into one long line, so if I write it line by line:

code:

In [33]: for row in query[sf.records:][0]: print row
AccountHistory

001000000072DKCAA2
00500000006wKQqAAM
2013-10-03T12:39:52.000Z
BillingCity
Nicoma Park

Which is what I'd like written as each list item. Make sense?

edit:

SurgicalOntologist posted:

I also have to ask... why are you doing int(str(total))? I can't think of any conceivable situation in which you'd want to convert something to a string and then to an integer.

Because I'm "bad at Python," and I couldn't think of any other way to handle type instance:

code:

In [36]: total
Out[36]: <size>...</size>

In [37]: str(total)
Out[37]: '37477'

In [38]: int(str(total))
Out[38]: 37477

the fucked around with this message at 17:49 on Jun 17, 2014

# ? Jun 17, 2014 17:46

SurgicalOntologist: Jun 17, 2004

Oh okay, strange, I would look at the documentation for the module you're working with, they might offer a more Pythonic way to convert to types. But given what you showed that actually makes some sense.

Judging by your post...

Python code:

records = query[sf.records:]  # some collection of records
# All I mean by collection is we can index into it,
# Presumably we can also loop over it
# or use it in things like str.join that take iterables.

# Each record consists of rows
# record = records[0]  #  if we wanted just the first one
# Then we could
# long_record = ', '.join(str(row) for row in record)

# Presumably we want all of them though.
long_records = [', '.join(str(row) for row in record)
                for record in records]

Try that, and play around with the stuff in comments too.

Edit: Something else that might help you keep everything straight is to refactor your procedure into small functions. Write a function that converts a single record to a single string (similar to the above), maybe 2-3 lines. Then write a another function that converts a query to a list of strings (each corresponding to one record). Your "main" function or script then can call these functions and it will look very clean.

SurgicalOntologist fucked around with this message at 18:00 on Jun 17, 2014

# ? Jun 17, 2014 17:54

the: Jul 18, 2004; by Cowcaster

Thanks, I'm still at a novice stage with handling strings and data in Python, so this stuff is all a bit new to me. I come from a Physics background, so my knowledge is mainly in working with the Numpy/Scipy modules and doing high level math applications. It's very frustrating trying to do simple things like storing data and hitting walls on it, but I'm trying!

On that note, the documentation looks a little like a Foreign language to me, because I don't come from a computer science/programming background. What should I be looking for, exactly?

edit: \/\/

the fucked around with this message at 18:11 on Jun 17, 2014

# ? Jun 17, 2014 17:58

SurgicalOntologist: Jun 17, 2004

That's not the documentation, that's the source code. It doesn't appear to have any documentation beyond the examples on the website.

# ? Jun 17, 2014 18:02

the: Jul 18, 2004; by Cowcaster

Well, this isn't the most Pythonic solution, but it works:

Python code:

query = svc.query("SELECT AccountId, CreatedById, 
CreatedDate, Field, NewValue, OldValue FROM AccountHistory WHERE 
Account.Industry = \'Government\' AND Account.Type = \'Prospect\'")


for rec in query[sf.records:]:
	templist = []
	for i in range(0,len(rec),1):
		templist.append(str(rec[i]))
	list1.append(templist)


total = query['size']
queryL = query['queryLocator']

while str(query['done']) == 'false' and len(list1) < int(str(total)):
	query = svc.queryMore(str(queryL))
	if (len(list1)+500) < int(str(total)):
		queryL = query['queryLocator']
	for rec in query[sf.records:]:
		templist = []
		for i in range(0,len(rec),1):
			templist.append(str(rec[i]))
		list1.append(templist)

# ? Jun 17, 2014 19:09

QuarkJets: Sep 8, 2008

Does your database really store 'false' as a string instead of a bool?

# ? Jun 17, 2014 19:26

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

QuarkJets posted:

Does your database really store 'false' as a string instead of a bool?

No, it's a bool. The string conversion & comparison is entirely unnecessary there. (i'm helping him out over in the databases thread)

fletcher fucked around with this message at 19:32 on Jun 17, 2014

# ? Jun 17, 2014 19:30

the: Jul 18, 2004; by Cowcaster

Well, I tried while query['done'] is False and it wasn't working so.... \/ :sigh:

# ? Jun 17, 2014 19:52

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

the posted:

Well, I tried while query['done'] is False and it wasn't working so.... \/\/

Using query['done'] is probably 'more correct' than my if query['queryLocator'] that I had used in the other thread:

code:

query = svc.query("SELECT Id, Name FROM Account")

all_records = []

if query['size'] > 0:
    done = False
    while not done:
        all_records = all_records + query['records']
        if not query['done']:
            query = svc.queryMore(query['queryLocator'])
        else:
            done = True

for record in all_records:
    print('%s, %s, %s' % (record.type, record.Id, record.Name))

# ? Jun 17, 2014 19:56

JHVH-1: Jun 28, 2002

Is there a decent IDE out there for windows that would let me work on python code from a remote machine. Right now I just use WinSCP, then edit the file in windows VIM and run it from putty. Then I also have to do all the git checkins/outs.

It works but I don't know if there is a nicer way and maybe something more full featured.

# ? Jun 17, 2014 19:59

OnceIWasAnOstrich: Jul 22, 2006

The commercial version of PyCharm supports both deployment to a remote server via SSH or SFTP as a well as remote interpreters and remote debugging. Unfortunately the free community version does not.

Alternatively just having git push/pulls shouldn't be that big of a deal. Before I had deployment and remote execution set up I just git pushed stuff to remote and then git pull/ran the script. I had this all in a 4-5 line Powershell script so I could do it in one command from my home Windows machine.

OnceIWasAnOstrich fucked around with this message at 20:15 on Jun 17, 2014

# ? Jun 17, 2014 20:11

JHVH-1: Jun 28, 2002

Maybe I can convince work to pay for PyCharm. I can live with the way I am doing it now, but the extra features would be nice.

Its all tied to specific networks and python setups I am locked in with on the servers, otherwise I would have more flexibility.

# ? Jun 17, 2014 20:39

the: Jul 18, 2004; by Cowcaster

if I do something like

Python code:

for row in mylist:
     if such and such is true:
          mylist.append(blahblah)

Is that just going to endlessly loop forever?

# ? Jun 17, 2014 21:33

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

the posted:

if I do something like
Python code:
for row in mylist:
     if such and such is true:
          mylist.append(blahblah)
Is that just going to endlessly loop forever?

It might. It depends upon how often "such and such" is true.

Assuming that you're appending one item to the list every time through, and assuming that "such and such" doesn't have any side effects (in particular, it doesn't modify the list): every time "such and such" is true, the number of items remaining to be "visited" stays the same, because you've advanced one place in the list but also lengthened the list by one. Every time it's false, the number of items remaining to be "visited" decreases by one, because you've advanced one place in the list without lengthening it. Once "such and such" has been false sufficiently many times (as many times as there initially were items in the list, to be precise), the loop will end.

# ? Jun 17, 2014 21:45

OnceIWasAnOstrich: Jul 22, 2006

Assuming that you don't want to test those conditions for items added during the loop, what you want to do is add your new items to a temporary list and then extend mylist

Python code:

templist = []
for row in mylist:
     if such and such is True:
          templist.append(blahblah)
mylist.extend(templist)

Also that boolean statement...

OnceIWasAnOstrich fucked around with this message at 21:59 on Jun 17, 2014

# ? Jun 17, 2014 21:56

SurgicalOntologist: Jun 17, 2004

the posted:

if I do something like
Python code:
for row in mylist:
     if such and such is true:
          mylist.append(blahblah)
Is that just going to endlessly loop forever?

Get in the habit of making functions, thereby allowing you to use concise list comprehensions.

Python code:

def convert_row_to_blahblah(foo):
    ...
    return blahblah


def such_and_such(foo):
    ...
    return True


new_list = [convert_row_to_blahblah(row) for row in mylist if such_and_such(row)]
mylist.extend(new_list)

# ? Jun 17, 2014 22:16

Haystack: Jan 23, 2005

JHVH-1 posted:

Maybe I can convince work to pay for PyCharm. I can live with the way I am doing it now, but the extra features would be nice.

Its all tied to specific networks and python setups I am locked in with on the servers, otherwise I would have more flexibility.

PyCharm is excellent, and you should definitely try to get a license. But if you can't, Aptana Studio is an acceptable free substitute. It's built on Eclipse so it'll do nearly anything if you cajole it enough.

# ? Jun 17, 2014 22:23

JHVH-1: Jun 28, 2002

Haystack posted:

PyCharm is excellent, and you should definitely try to get a license. But if you can't, Aptana Studio is an acceptable free substitute. It's built on Eclipse so it'll do nearly anything if you cajole it enough.

I was actually trying to set up Eclipse at work with pydev or whatever, but it was a big pain in the butt and I got fed up. I used to use Eclipse in college for Java and was looking for something like that. Thanks for the suggestion.

# ? Jun 17, 2014 23:53

Haystack: Jan 23, 2005

It's still pydev at the core, mind you. It's just preconfigured in a nice package.

# ? Jun 18, 2014 00:01

Akarshi: Apr 23, 2011

Okay guys, I'm super new to Python and I think this question is Python related. If it's not, though, feel free to point me in the right place.

I'm trying to figure out how to compare the contents of two Google Protocol Buffer messages. So I have these two messages, and they have a lot of different values that sometimes drill down to more values (for example, I have a Message that has a string, an int, and a custom_snapshot, where custom_snapshot is comprised of an int, a string, and so on). I want to see if these two messages are the same. I don't want to compare each value one by one, since that will take a while, so I was wondering if there was a quick way to do this in Python?

I tried doing messageA.debugString() == messageB.debugString(), but apparently there is no debugString method that I could access when I tried. Any help would be appreciated. Thanks!

# ? Jun 19, 2014 00:04

evilentity: Jun 25, 2010

Akarshi posted:

Okay guys, I'm super new to Python and I think this question is Python related. If it's not, though, feel free to point me in the right place.

I'm trying to figure out how to compare the contents of two Google Protocol Buffer messages. So I have these two messages, and they have a lot of different values that sometimes drill down to more values (for example, I have a Message that has a string, an int, and a custom_snapshot, where custom_snapshot is comprised of an int, a string, and so on). I want to see if these two messages are the same. I don't want to compare each value one by one, since that will take a while, so I was wondering if there was a quick way to do this in Python?

I tried doing messageA.debugString() == messageB.debugString(), but apparently there is no debugString method that I could access when I tried. Any help would be appreciated. Thanks!

You sure you cant just do messageA == messageB?
If you have custom messages you need to implement __eq__ magic method.

# ? Jun 19, 2014 00:20

Akarshi: Apr 23, 2011

evilentity posted:

You sure you cant just do messageA == messageB?
If you have custom messages you need to implement __eq__ magic method.

I can do that? I thought that since the two messages are from different places, but should have the same content, I wouldn't be able to do that. If I can though, that'd be great.

Also, what is __eq__? Is it like a comparator in Java?

# ? Jun 19, 2014 00:28

Blinkz0rz: May 27, 2001; MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Akarshi posted:

I can do that? I thought that since the two messages are from different places, but should have the same content, I wouldn't be able to do that. If I can though, that'd be great.

Also, what is __eq__? Is it like a comparator in Java?

__eq__() is the functional equivalent of ==. Calling a == b is the same as a.__eq__(b) && b.__eq__(a)

# ? Jun 19, 2014 01:07

FoiledAgain: May 6, 2007

Akarshi posted:

Also, what is __eq__? Is it like a comparator in Java?

Here's an example that might help. I'm have Word objects that have both a spelling attribute and a pronunciation attribute. In the particular work that I'm doing, I never care if two words are spelled the same, I only care if they are pronounced the same. It is convenient if I can find this out by writing word1==word2 in my code so I've used __eq__ as follows:

code:

class Word(object):

    def __init__(self, spelling, pronunciation):
        self.spelling = spelling
        self.pronunciation = pronunciation

    def __eq__(self, other):
        return self.pronunciation == other.pronunciation

# ? Jun 19, 2014 01:16

JHVH-1: Jun 28, 2002

If you are using a standard format, it looks like there is a python module set up to handle those with its own message class https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.message.Message-class

Maybe it would be cleaner. I had to do some stuff similar a few times, like pulling data out of a database and an xml tag may not exist when you try to compare it so you get an error. I think I ended up checking somehow if it exists and then compared it. I forget how but I got around having a try/except block by using a built in class for the type of data I was using. Sometimes re-thinking how you store the data inside your code ends up making it easier to deal with and read.

# ? Jun 19, 2014 01:40

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 05:12

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Blinkz0rz posted:

__eq__() is the functional equivalent of ==. Calling a == b is the same as a.__eq__(b) && b.__eq__(a)

This is not true. a == b is just a.__eq__(b)

You shouldn't do this but __eq__ doesn't need to be symmetric.

code:

>>> class ShittyInt(int):
...     def __eq__(self, other):
...             if self > other:
...                     return False
...             else:
...                     return True
...
>>> a = ShittyInt(4)
>>> b = ShittyInt(8)
>>> a == b
True
>>> b == a
False

# ? Jun 19, 2014 12:59

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »