Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

FoiledAgain: May 6, 2007

OK, so three matplotlib suggestions in a row. I'll go for it. After some browsing through the gallery, I think I can get what I want using something simple like this, plus maybe some subplotting. Thanks thread!

# ? Apr 22, 2013 23:23

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 22:41

Dominoes: Sep 20, 2007

Dren posted:

Dominoes, you were talking about having data from two sources. One dataset is in json the other is in csv. Do the two data sources provide you the same data? If so, you should look at creating your own type, call it StockData. When you ingest data, load it into a StockData object. The custom code for how to read in each type of data will go into those load functions. Your code will access the StockData object. This way you don't have to write nutty custom accessor stuff (like the code you have).

This is just a suggestion, you might have a need for the way your stuff is right now.

Correct. I'm pulling real-time, and historical data from both sources. The code I posted is part of the historical data section. The real-time functions are much cleaner, since the sources let me pull data from 200 and 2500 stocks at once, so I don't have to loop through each stock. Results should be the same for each source. I don't see why that wouldn't work - I'll try to do it.

I finished my first computer program, and sent it to my coworkers tonght! It provides an interface for quickly downloading the latest version of a selection of FAA instrument procedures, and saving them as a combined PDF, or individual ones. I'm hoping to make it universal later, with the ability to select any FAA procedure, instead of just the ones I loaded.

Dominoes fucked around with this message at 02:30 on Apr 23, 2013

# ? Apr 22, 2013 23:50

Rohaq: Aug 11, 2006

So I'm writing a Django app that needs to talk to our RequestTracker system: No problem, the python-rtkit module does a pretty good job of talking to the RT API.

Except I found that it uses urllib2 and the BasicAuthenticator (the only method I have of connecting to our current RT setup), and when there's a a bad username/password, urllib2 goes into a recursive loop:

code:

Exception Type: 	RuntimeError
Exception Value: 	maximum recursion depth exceeded
Exception Location: 	/usr/lib/python2.6/urllib2.py in do_open, line 1118

I mean, I don't plan on having bad login credentials in production, but on the offchance the service account is disabled some day, I'd like to return a useful error message regarding the fact.

Can anyone suggest a good way to catch the bad login the first time around, rather than letting urllib2 go into its loop?

# ? Apr 23, 2013 15:49

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Rohaq posted:

So I'm writing a Django app that needs to talk to our RequestTracker system: No problem, the python-rtkit module does a pretty good job of talking to the RT API.

Except I found that it uses urllib2 and the BasicAuthenticator (the only method I have of connecting to our current RT setup), and when there's a a bad username/password, urllib2 goes into a recursive loop:
code:
Exception Type: 	RuntimeError
Exception Value: 	maximum recursion depth exceeded
Exception Location: 	/usr/lib/python2.6/urllib2.py in do_open, line 1118
I mean, I don't plan on having bad login credentials in production, but on the offchance the service account is disabled some day, I'd like to return a useful error message regarding the fact.

Can anyone suggest a good way to catch the bad login the first time around, rather than letting urllib2 go into its loop?

What you're doing shouldn't be necessary and suggests you have a series of bad redirects on the server side that's causing this loop in the first place. Fire up Wireshark and see what's going on.

Vulture Culture fucked around with this message at 16:17 on Apr 23, 2013

# ? Apr 23, 2013 16:15

Rohaq: Aug 11, 2006

Misogynist posted:

What you're doing shouldn't be necessary and suggests you have a series of bad redirects on the server side that's causing this loop in the first place. Fire up Wireshark and see what's going on.

Quite likely, our RequestTracker setup has been hacked to pieces over the years. We're supposedly getting it upgraded at some point soon.

I'll fire up wireshark at some point, though at the moment I'm just trying to dig through the module using dir to try and figure out the easiest way to get the Requestor out of a ticket.

# ? Apr 23, 2013 16:34

Smarmy Coworker: May 10, 2008; by XyloJW

Questions since Twisted makes no sense to me, at all:

Is the reactor system-wide or process-wide? Can I perhaps run two different TCP connections at the same time, as separate programs?

task.LoopingCall is a Deferred so I can add callbacks, but the documentation says nothing about how they're processed other than "it will run callbacks when it stops." Does the callback happen every loop?

Fun note:

Twisted FAQ posted:

Why can't the Twisted's reactor be restarted?

There is no interesting reason. This is mainly a limitation of the current implementation, and a lack of interest on this particular feature. It's just a matter of someone spending the time on it.

# ? Apr 23, 2013 18:36

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

ARACHNOTRON posted:

Questions since Twisted makes no sense to me, at all:

Is the reactor system-wide or process-wide?

The reactor is process-wide. It wouldn't make sense otherwise.

ARACHNOTRON posted:

task.LoopingCall is a Deferred so I can add callbacks, but the documentation says nothing about how they're processed other than "it will run callbacks when it stops." Does the callback happen every loop?

The callback happens however often you specify when you call start().

ARACHNOTRON posted:

Fun note:

It's a hard problem that's been tried to be solved multiple times by multiple people, with limited success.

# ? Apr 23, 2013 19:01

Smarmy Coworker: May 10, 2008; by XyloJW

That would be a good explanation to have in the FAQ instead of "no reason, just nobody cares."

Reactor thing makes sense, but maybe I'm super confused about what I even want to do w/r/t the LoopingCall question.

Say I have a message queue with something consuming it. Separate thread and everything.

code:

x = LoopingCall( reactor.callInThread, dequeue )

dequeue pops messages off an internal storage queue and returns the message. I'm not really sure how to get that return value so I thought I might add another callback that somehow handled this one, except now I'm not sure about that either!

Considering creation of these Deferred object is (callable, *args, **kargs) and I had a weird chain like above, if I did:

code:

x = LoopingCall( reactor.callInThread, dequeue, otherfunc )
y = Loop: Call: dequeue(otherfunc)

print x == y ? ':)' : ':('

edit
Well, I decided to just do this:

Python code:

# ~ module TR~
# ...
def pull_from_queue():
	reactor.callInThread(tw.dequeue, sendMessage)

x = LoopingCall(pull_from_queue)
# ...

# ~ module tw ~
# ...
def dequeue(func):
	addr = '1-888-888-8888'
	try:
		msg = q.get_nowait()
		q.task_done()
	except Empty:
		msg = None
	finally:
		func(addr, msg)

how bad is it

Smarmy Coworker fucked around with this message at 20:30 on Apr 23, 2013

# ? Apr 23, 2013 19:34

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Don't use threads with Twisted. It is not going to go well. callInThread is designed for a worker task like model where you have a synchronous blocking API that you need to make asynchronous (like DBAPI).

# ? Apr 23, 2013 21:11

Smarmy Coworker: May 10, 2008; by XyloJW

That is actually literally the case. RabbitMQ/pika message queue consumption :/

Should I instead have it call channel.start_consuming ?

or, wait, sorry, I start the consumption process itself as a separate thread, and I don't know why I'm using callInThread for a non-threaded function, wow what the heck?? :confused:

I think it was because python.Queue is synchronous/blocking

Smarmy Coworker fucked around with this message at 21:44 on Apr 23, 2013

# ? Apr 23, 2013 21:28

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

ARACHNOTRON posted:

That is actually literally the case. RabbitMQ/pika message queue consumption :/

Before anyone else chimes in I'd like to point out that while there is an AMQP library for Twisted, its performance is really, really bad.

# ? Apr 24, 2013 02:42

Tiax Rules All: Jul 22, 2007; You are but the grease for the wheels of his rule.

https://github.com/mikemaccana/python-docx

I've been playing around with this module to do basic stuff with Word docx files. Right now I'm having some trouble with the savedocx function in the main docx.py module. I know what arguments I should be passing for document and output, but all the extra parameters for this function are confusing me. I just want to save my new file.

# ? Apr 25, 2013 05:27

tef: May 30, 2004; -> some l-system crap ->

the test code has an example of calling savedoc https://github.com/mikemaccana/python-docx/blob/master/tests/test_docx.py

# ? Apr 25, 2013 06:15

Dren: Jan 5, 2001; Pillbug

Does anyone know of gui code for displaying/editing an arbitrary dictionary? It would be nice if it is easy to extend it to allow displaying/editing the fields of arbitrary classes.

# ? Apr 25, 2013 17:07

Smarmy Coworker: May 10, 2008; by XyloJW

Well hey! Turns out Pika has an adapter for Twisted along with Tornado and some others. If anybody is in my situation and doesn't know anything, like me, hope this maybe helps a little bit??

Dren posted:

Does anyone know of gui code for displaying/editing an arbitrary dictionary? It would be nice if it is easy to extend it to allow displaying/editing the fields of arbitrary classes.

Are you asking for code that allows you to display and edit a dictionary from inside a GUI? That exists already? Just to be slightly more clear.

That's probably something you'd have to write yourself but I could maybe do up some pseudocode after I'm done smashing my face against a wall, if you need it.

# ? Apr 25, 2013 17:31

Dren: Jan 5, 2001; Pillbug

ARACHNOTRON posted:

Are you asking for code that allows you to display and edit a dictionary from inside a GUI? That exists already? Just to be slightly more clear.

Yep, but only if it exists already. If not, don't sweat it. I'll either avoid writing it or quickly throw something together. I guess I would prefer pyQT or pyGTK. I don't much like tkinter.

# ? Apr 25, 2013 17:37

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Why would that be helpful? As a debugging tool? There are already some excellent GUI Python debuggers.

# ? Apr 25, 2013 18:03

Dren: Jan 5, 2001; Pillbug

Suspicious Dish posted:

Why would that be helpful? As a debugging tool? There are already some excellent GUI Python debuggers.

I am writing something to parse out a jpeg file into its markers and their data. I will parse some of the metadata associated with the markers but skip parsing of image data like huffman tables and DCT coefficients. After I parse it all out (probably into a dict) I'm going to make some modifications to it and write it back to a new file without doing any validation or anything (it can write invalid jpeg files).

It would be nice to be able to visually edit the data. I figured someone might have written something to display and edit arbitrary python data.

Now that I'm thinking about it more I suppose I could serialize to json, edit with a text editor, then use the modified json to rewrite the file. json serialization has hooks for serializing custom objects too, which is nice. The binary blobs would be kind of ugly but it'd be easier than dealing with GUI toolkits.

# ? Apr 25, 2013 18:23

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

You can't really ask some user to edit "a dictionary". There could be all sorts of arbitrary objects as the keys and values in the dictionary, and you'd have to make a custom editor widget for each one. Even if you stick to just "plain" things (string keys, JSON-like values) it's not really a good UI in general.

The JSON file sounds like a much more reasonable approach, but I wouldn't use the custom reader/writer stuff in there. Just write some method to go to a dictionary, and a classmethod to load from a dictionary, and then run json.dumps() over the result of serialization, and json.loads() over the loading.

Also, are you trying to make large.jpg or something?

# ? Apr 25, 2013 18:36

Dren: Jan 5, 2001; Pillbug

Suspicious Dish posted:

You can't really ask some user to edit "a dictionary". There could be all sorts of arbitrary objects as the keys and values in the dictionary, and you'd have to make a custom editor widget for each one. Even if you stick to just "plain" things (string keys, JSON-like values) it's not really a good UI in general.

The JSON file sounds like a much more reasonable approach, but I wouldn't use the custom reader/writer stuff in there. Just write some method to go to a dictionary, and a classmethod to load from a dictionary, and then run json.dumps() over the result of serialization, and json.loads() over the loading.

Also, are you trying to make large.jpg or something?

I agree with you about the GUI idea. I think it could've worked if I stuck to the "plain" values you suggested (which I would've done). However it would've been a lot of effort for a crude result. The json approach seems much cleaner so I'll go with that.

Why wouldn't you use the json module's custom reader/writer stuff? I don't much like the json module's approach to custom encoding/decoding -- subclassing the module's encoder object seems crude compared to an approach of coupling the serialization to the classes. However, it's not prohibitively bad.

I'm trying to create purposefully broken jpegs to test some routines that validate jpeg marker values. What's large.jpg?

# ? Apr 25, 2013 18:55

Experto Crede: Aug 19, 2008; Keep on Truckin'

Does anyone know what the hexversion output of python 2.7.0 is?

# ? Apr 25, 2013 21:45

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

0x207000F0. Why do you care, though?

# ? Apr 25, 2013 21:56

Experto Crede: Aug 19, 2008; Keep on Truckin'

Suspicious Dish posted:

0x207000F0. Why do you care, though?

Need to run different code dependent on whether the version is <2.7

# ? Apr 25, 2013 22:02

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

First off, I don't believe you. You should test if things are available, not the version. Second, if you must, use sys.version_tuple < (2, 7)

# ? Apr 25, 2013 22:34

ufarn: May 30, 2009

I just stumbled on this notation:

Python code:

foo == 1 or 2 or 3

Is this idiomatic Python, and does it even do what I think it does? Because it'd make for some prettier code in some of my projects.

# ? Apr 28, 2013 20:09

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

It does not do what you think it does, and if the person wrote it like that, it's a bug. Thankfully, there's the awesome alternative: foo in (1, 2, 3).

# ? Apr 28, 2013 20:33

QuarkJets: Sep 8, 2008

When you're done with an object that takes up a lot of memory (for instance, a 1k by 1k by 1k numpy array), is it considered better practice to delete it with del or just leave it for garbage collection?

# ? Apr 28, 2013 21:07

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

QuarkJets posted:

When you're done with an object that takes up a lot of memory (for instance, a 1k by 1k by 1k numpy array), is it considered better practice to delete it with del or just leave it for garbage collection?

In terms of GC, del doesn't do anything but remove a reference to the object. If it's about to go out of scope already anyway, it's pointless to explicitly call it unless you're deliberately breaking a cycle somewhere.

Objects in Python don't have any defined destructor semantics and inherit them from the runtime, which makes object destruction sort of an onerous thing. In CPython objects are reference-counted and __del__ is called as soon as the last reference goes out of scope, but in IronPython and Jython they pick up the finalizer semantics from the CLR and JRE mark-and-sweep GCs, respectively.

# ? Apr 28, 2013 21:20

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

"del foo", in terms of what happens to the value of "foo", is the exact same thing as "foo = None" or "foo = 'asdf'".

It does not call __del__ or anything like that. Do not use __del__.

# ? Apr 28, 2013 21:21

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Misogynist posted:

Objects in Python don't have any defined destructor semantics and inherit them from the runtime, which makes object destruction sort of an onerous thing. In CPython objects are reference-counted and __del__ is called as soon as the last reference goes out of scope, but in IronPython and Jython they pick up the finalizer semantics from the CLR and JRE mark-and-sweep GCs, respectively.

CPython has a cycle collector too which means that destruction isn't deterministic if you have a reference cycle. I do not advocate using __del__ for anything as it can gently caress up your program in various, subtle ways.

# ? Apr 28, 2013 21:22

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

Suspicious Dish posted:

CPython has a cycle collector too which means that destruction isn't deterministic if you have a reference cycle. I do not advocate using __del__ for anything as it can gently caress up your program in various, subtle ways.

Cycle collecting itself isn't even deterministic if I recall correctly, because if any of the objects has a __del__ defined, the runtime goes "I don't know the right order to clean these up in, so gently caress it" and just leaves them around in perpetuity.

# ? Apr 28, 2013 21:29

Opinion Haver: Apr 9, 2007

Suspicious Dish posted:

It does not do what you think it does, and if the person wrote it like that, it's a bug. Thankfully, there's the awesome alternative: foo in (1, 2, 3).

To expand, what it will actually do is evaluate as

Python code:

(foo == 1) or 2 or 3

which will obviously always be True.

# ? Apr 28, 2013 21:36

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

yaoi prophet posted:

To expand, what it will actually do is evaluate as
Python code:
(foo == 1) or 2 or 3
which will obviously always be True.

It will be either True (if indeed foo == 1) or 2 (which is a "true" value in a Boolean context) (otherwise).

# ? Apr 28, 2013 23:05

Popper: Nov 15, 2006

Hammerite posted:

It will be either True (if indeed foo == 1) or 2 (which is a "true" value in a Boolean context) (otherwise).

Even if foo is not 1 it will return True, it's completely useless.

in on the other hand is the keyword you want.

# ? Apr 29, 2013 13:01

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

Popper posted:

Even if foo is not 1 it will return True, it's completely useless.

in on the other hand is the keyword you want.

No, you have not understood what I said. If foo != 1 then it will return 2, which is not True, although it is a "true" value in a Boolean context.

code:

Python 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> x = 4
>>> y = (x == 1) or 2 or 3
>>> y
2
>>> y is True
False
>>> x = 1
>>> y = (x == 1) or 2 or 3
>>> y
True
>>> y is True
True
>>>

# ? Apr 29, 2013 15:32

OnceIWasAnOstrich: Jul 22, 2006

Hammerite posted:

No, you have not understood what I said. If foo != 1 then it will return 2, which is not True, although it is a "true" value in a Boolean context.

To add to this although in this case this was probably just wrong, I have not infrequently used similar constructs so that I can return either False or a number where even zero represents a good value if I use is False.

# ? Apr 29, 2013 15:42

Lysidas: Jul 26, 2002; John Diefenbaker is a madman who thinks he's John Diefenbaker.; Pillbug

OnceIWasAnOstrich posted:

To add to this although in this case this was probably just wrong, I have not infrequently used similar constructs so that I can return either False or a number where even zero represents a good value if I use is False.

None is probably a better singleton for this:

http://docs.python.org/3/library/constants.html#None posted:

None

The sole value of the type NoneType. None is frequently used to represent the absence of a value, as when default arguments are not passed to a function. Assignments to None are illegal and raise a SyntaxError.

I'd be wary of using False since you can accidentally do math on it (unlike None):

code:

In [1]: False == 0
Out[1]: True

In [2]: False + 4
Out[2]: 4

# ? Apr 29, 2013 16:10

OnceIWasAnOstrich: Jul 22, 2006

Lysidas posted:

I'd be wary of using False since you can accidentally do math on it (unlike None):

Yeah, as I was writing that I was thinking I might be creating my own coding horror.

Although in my opinion it is dumb as poo poo that False and True are actually 0 and 1 for math-related purposes. Why would you ever actually want to do that? (I'm sure there is a reason and I'll probably learn another thing about Python from the answer.)

# ? Apr 29, 2013 16:35

Lysidas: Jul 26, 2002; John Diefenbaker is a madman who thinks he's John Diefenbaker.; Pillbug

I was curious about this myself -- it doesn't seem very consistent with the rest of Python's strict typing. Numerical arithmetic on booleans seems to come from a combination of historical reasons, ease of implementation, and the desire to be consistent with the type-coercing nature of the if and while constructs.

The True and False constants were added in Python 2.2.1 -- at the time these were just other ways to refer to the values 1 and 0. An actual boolean class was added in 2.3 and the docs are clear that it was a deliberate decision to allow arithmetic on booleans:

http://docs.python.org/3/whatsnew/2.3.html#pep-285-a-boolean-type posted:

Python's Booleans were added with the primary goal of making code clearer. For example, if you're reading a function and encounter the statement return 1, you might wonder whether the 1 represents a Boolean truth value, an index, or a coefficient that multiplies some other quantity. If the statement is return True, however, the meaning of the return value is quite clear.

Python's Booleans were not added for the sake of strict type-checking. A very strict language such as Pascal would also prevent you performing arithmetic with Booleans, and would require that the expression in an if statement always evaluate to a Boolean result. Python is not this strict and never will be, as PEP 285 explicitly says. This means you can still use any expression in an if statement, even ones that evaluate to a list or tuple or some random object. The Boolean type is a subclass of the int class so that arithmetic using a Boolean still works.

PEP 285 posted:

4) Should we strive to eliminate non-Boolean operations on bools in the future, through suitable warnings, so that for example True+1 would eventually (in Python 3000) be illegal?

=> No.

There's a small but vocal minority that would prefer to see "textbook" bools that don't support arithmetic operations at all, but most reviewers agree with me that bools should always allow arithmetic operations.

# ? Apr 29, 2013 16:59

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 22:41

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

There are very few cases for bools as arithmetic, but one of the old hacks for the lack of a ternary operator is one of them: foo = 3; bar = ['a', 'b'][foo == 3]

# ? Apr 29, 2013 17:19

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »