Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Digital Spaghetti: Jul 8, 2007; I never gave a reach-around to a spider monkey while reciting the Pledge of Alligence.

chemosh6969 posted:

I'm guessing xml.dom isn't installed or it's not on the path, since it's giving an import error.

Yea, seems to be fine since switching into linux. Windows sucks :|

# ? Nov 30, 2008 01:04

Adbot: ADBOT LOVES YOU

# ? May 24, 2024 01:27

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

It looks like you're importing your own version of xml.dom.minidom, instead of the one packaged with Python. Your version is attempting to import PyExpat, which is not packaged with Google Appengine.

You could try removing your xml/dom/minidom.py file and its parent directories. Google may have implemented their own DOM library that does not depend on a C DLL.

# ? Nov 30, 2008 02:21

MononcQc: May 29, 2007

Milde posted:

Unicode isn't an encoding, it's a more general standard that specifies many different encodings, character sets, collations, etc. Your database is returning a latin-1-encoded string, not a UTF-8 string. You should check to see if your database's charset is set to UTF-8, and that you're instantiating your MySQLdb connection with charset='utf-8' and use_unicode=True as part of the arguments (to MySQLdb.connect()).

I have tried both of these things before and it still doesn't work. I don't know, I think I'll just give up using utf8 for a collation/charset and go back to latin-1 databases, columns and tables. I just don't see the benefits after losing so much time for no reason.

# ? Nov 30, 2008 21:13

kaschei: Oct 25, 2005

I've been googling for a little less than an hour for a Python module that can parse functions and evaluate them. I haven't found anything like that, and I'm looking for something that can handle multiple (at least two) variables.

Ideally I could prompt for keyboard input, they could type in "4(x^2)*y + x*sin(y)" and I would get an object that I could cause to evaluate at given values, like:

code:

>>> fn = functionParse(raw_input("f(x,y) = ")
f(x,y) = 4(x^2)*y + x*sin(y)
>>> fn.eval(.5,Math.pi)
3.1415926535897931

# ? Nov 30, 2008 21:29

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

MononcQc posted:

I have tried both of these things before and it still doesn't work. I don't know, I think I'll just give up using utf8 for a collation/charset and go back to latin-1 databases, columns and tables. I just don't see the benefits after losing so much time for no reason.

Did you convert all of the data in the database from latin-1 to utf-8? Since querying the database still returns latin-1 strings, I'd say that's the most likely cause of your problem.

http://yoonkit.blogspot.com/2006/03/mysql-charset-from-latin1-to-utf8.html details how to do the conversion.

# ? Nov 30, 2008 21:51

tripwire: Nov 19, 2004; _{ghost flow}

kaschei posted:

I've been googling for a little less than an hour for a Python module that can parse functions and evaluate them. I haven't found anything like that, and I'm looking for something that can handle multiple (at least two) variables.

Ideally I could prompt for keyboard input, they could type in "4(x^2)*y + x*sin(y)" and I would get an object that I could cause to evaluate at given values, like:
code:
>>> fn = functionParse(raw_input("f(x,y) = ")
f(x,y) = 4(x^2)*y + x*sin(y)
>>> fn.eval(.5,Math.pi)
3.1415926535897931

Pyparsing would work pretty well for this; theres even examples that do sort of almost what you want, it wouldn't take much work to extend it more.

# ? Nov 30, 2008 23:42

MononcQc: May 29, 2007

Habnabit posted:

Did you convert all of the data in the database from latin-1 to utf-8? Since querying the database still returns latin-1 strings, I'd say that's the most likely cause of your problem.

http://yoonkit.blogspot.com/2006/03/mysql-charset-from-latin1-to-utf8.html details how to do the conversion.

No, in fact the old latin-1 entries come out right, the new utf-8 ones are those that get messed up. I did the thing in that post (converting the database, table, each column). Next thing I'll try will be to just start with a new database right in utf8, as I figure the problem must be what I did rather than the language.

# ? Nov 30, 2008 23:52

king_kilr: May 25, 2007

kaschei posted:

I've been googling for a little less than an hour for a Python module that can parse functions and evaluate them. I haven't found anything like that, and I'm looking for something that can handle multiple (at least two) variables.

Ideally I could prompt for keyboard input, they could type in "4(x^2)*y + x*sin(y)" and I would get an object that I could cause to evaluate at given values, like:
code:
>>> fn = functionParse(raw_input("f(x,y) = ")
f(x,y) = 4(x^2)*y + x*sin(y)
>>> fn.eval(.5,Math.pi)
3.1415926535897931

if you consider it safe to do a raw eval() you could use SymPy

# ? Dec 1, 2008 08:56

ahobday: Apr 19, 2007

I was looking at GUI programming with Python, but I figured I'd just ask here instead of wading around the internet trying to find an answer: Do any of the Python GUI doohickies work in the same way the visual programming languages work, like VB.net and C#? As in, you can just drag and drop GUI components and then put your code in behind the scenes, as it were?

# ? Dec 3, 2008 17:08

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

Centipeed posted:

I was looking at GUI programming with Python, but I figured I'd just ask here instead of wading around the internet trying to find an answer: Do any of the Python GUI doohickies work in the same way the visual programming languages work, like VB.net and C#? As in, you can just drag and drop GUI components and then put your code in behind the scenes, as it were?

None that I know of. Both GTK+ and Qt have drag-and-drop interface designers, but rather than generate code they create UI definition files which the programmer loads explicitly (if you've ever used Interface Builder on a Mac, same idea).

Gtk+ has Glade (win32 installer):

Click here for the full 1024x768 image.

Qt has Qt Designer:

Click here for the full 1413x996 image.

Either way you'll need to install the associated Python support, with PyGTK / PyGlade or PyQt

# ? Dec 3, 2008 18:29

yippee cahier: Mar 28, 2005

I'm facing a decision in a script I'm working on and would appreciate any insight or comments.

I'm turning raw bayer sensor data from an image acquisition system into basic DNG files suitable to be run through dcraw. I want to use disk space efficiently as these files will be archived for years and can take up an enormous amount of room. To do this, I want the pixel values stored in their native bit depth. I'm doing this all already, but repacking the data in 12-bits using my current method takes way too much time.

I have to do a lot of bit masking and shifting to repack the data. Right now my test code operates on an array.array of bytes. Fundamentally, I'm rearranging things on a nibble level, which is why I'm masking and shifting bytes. Maybe there is an efficient way of dealing with 4-bit values? I couldn't really find anything, so here are the options I can see:

rewrite it to process things in a few larger chunks, like int or long and get the processor to use the same instruction to process more samples
try numpy and see how it does with this sort of thing
learn how to write a C module and let that handle the dirty work
any other ideas?

Basically, can python efficiently shift and mask bits and I'm just going about it the wrong way, or would this sort of problem be a classic example of where not to use python? Any thoughts on these options?

Examples of operations I'm doing:
repacking 12-bit packed data into the expected order:

code:

for i in range(len(imgbuf)/3):
	temp = imgbuf[i*3+1]
	imgbuf[i*3+1] = ((imgbuf[i*3]&0xF)<<4) + ((imgbuf[i*3+2]&0XF0)>>4)
	imgbuf[i*3] = ((temp&0x0F)<<4) + ((imgbuf[i*3]&0xF0)>>4)
	imgbuf[i*3+2] = ((imgbuf[i*3+2]&0xF)<<4) + ((temp&0xF0)>>4)

packing padded 16-bit samples into 12-bit packed data in the expected order:

code:

for i in range(len(rawbuf)/4):
	imgbuf.append( (rawbuf[i*4+1]<<4) + (rawbuf[i*4]>>4) )
	imgbuf.append( ((rawbuf[i*4]&0xF)<<4) + rawbuf[i*4+3] )
	imgbuf.append( rawbuf[i*4+2] )

# ? Dec 4, 2008 00:39

JoeNotCharles: Mar 3, 2005; Yet beyond each tree there are only more trees.

If you're packing these just for disk space storage, why not just run them through gzip?

# ? Dec 4, 2008 01:03

yippee cahier: Mar 28, 2005

JoeNotCharles posted:

If you're packing these just for disk space storage, why not just run them through gzip?

The DNG format supports transparent lossless compression internally. It approaches file size I get when I properly pack the bits, but I can apply it on top of the properly packed bits for another 5-10% compression.

# ? Dec 4, 2008 01:48

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

performing a whole bunch of bitwise operations in a performance-critical loop is p. much the perfect example of when not to use Python.

Use numpy, or if it doesn't have the appropriate operations / is still too slow then C.

# ? Dec 4, 2008 01:53

Kire: Aug 25, 2006

Python 3.0 released, according to Slashdot!

# ? Dec 4, 2008 20:03

Janitor Prime: Jan 22, 2004; PC LOAD LETTER

What da fuck does that mean; Fun Shoe

Kire posted:

Python 3.0 released, according to Slashdot!

There is a Ptyon 3.0 thread already.

# ? Dec 4, 2008 20:10

Plastic Snake: Mar 2, 2005; For Halloween or scaring people.

Most likely there is a much easier way of doing what I want to do, but here goes.

I want to put the contents of string.punctuation:

code:

!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~

into a list, with each character being one entry in the list. I've tried using string.split, but can't find a way to just make it split every character apart because there is no delimiter in the string. I've noticed that string.split does allow you to tell it how many times to split the string, but it seems to be dependent on there being a delimiter. Any ideas?

# ? Dec 4, 2008 20:19

MononcQc: May 29, 2007

Plastic Snake posted:

Most likely there is a much easier way of doing what I want to do, but here goes.

I want to put the contents of string.punctuation:
code:
!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~
into a list, with each character being one entry in the list. I've tried using string.split, but can't find a way to just make it split every character apart because there is no delimiter in the string. I've noticed that string.split does allow you to tell it how many times to split the string, but it seems to be dependent on there being a delimiter. Any ideas?

would this work:

code:

list('''!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~''')

or am I understanding it wrong?

# ? Dec 4, 2008 20:30

Plastic Snake: Mar 2, 2005; For Halloween or scaring people.

Why yes, yes it would. I was aiming for being able to just use string.punctuation instead of the actual string, but that was just because i didn't realize list() even existed. I'm just starting to learn Python so I've still got a long ways to go. Thanks for the help!

# ? Dec 4, 2008 20:35

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

Plastic Snake posted:

Why yes, yes it would. I was aiming for being able to just use string.punctuation instead of the actual string, but that was just because i didn't realize list() even existed. I'm just starting to learn Python so I've still got a long ways to go. Thanks for the help!

list(string.punctuation)

# ? Dec 4, 2008 21:00

bitprophet: Jul 22, 2004; Taco Defender

Call me stupid, but why do you need it explicitly turned into a list given that strings already support indexing, slicing and iteration? :v:

(If you didn't know that -- well, they do

)

# ? Dec 4, 2008 21:47

Plastic Snake: Mar 2, 2005; For Halloween or scaring people.

bitprophet posted:

Call me stupid, but why do you need it explicitly turned into a list given that strings already support indexing, slicing and iteration? (If you didn't know that -- well, they do )

Even better! I've literally only been at this for maybe a week now, but I've tried to learn python a couple times before this. All the concepts and such are finally starting to click at a fairly decent pace, but I've just gotta dive in and do lots more of it to improve.

# ? Dec 4, 2008 23:50

tayl0r: Oct 3, 2002; This post brought to you by SOE

I'm making a web app in Google App Engine and I'm not having much luck serializing a Model object into a JSON string.
I'm using the simplejson library (it's what Google uses in their example code) and if I try to use the dumps() method on my Model object I get this error:

code:

TypeError: <adventureModel.Page object at 0x21d1410> is not JSON serializable

Which makes sense, since the Model object has some things in it that are specific to Google App Engine, and apparently simplejson hasn't been updated to know how to serialize/encode them.

I did some search online and came across this: http://stackoverflow.com/questions/212125/turning-a-gqlquery-result-set-into-a-python-dictionary

I'm doing exactly that in my own python code, but dumps() doesn't seem to be giving me back valid JSON.

If my model looks like this:

code:

class Page(db.Model):
	adventure = db.ReferenceProperty(Adventure)
	name = db.StringProperty()
	def toDict(self):
		return {
			'adventure': self.adventure.key().id(),
			'id': self.key().id(),
			'name': self.name,
		}

And here is my python code for creating the JSON:

code:

  pagesJSON = {}
  for page in pages:
    pagesJSON[page.key().id()] = page.toDict()
  self.response.out.write("[\n")
  self.response.out.write(simplejson.dumps(pagesJSON))
  self.response.out.write("\n]")

I get back this JSON:

code:

{"6": {"adventure": 5, "name": "page name", "id": 6}}

Notice there are no enclosing square brackets, which you need for valid JSON (I think). This JSON does not read into my javascript at all, so I'm assuming its not valid JSON.

I manually tested this JSON string, which does get parsed into my java script:

code:

[
  {"adventure": 5, "name": "page name", "id": 6}
]

So, here is my question: how the gently caress can I make this work?
* I need to manually enclose the entire JSON string with square brackets.
* I need to change my pagesJSON object to be more like an array instead of a dict/hash, so it just has a raw list of page objects inside it, instead of each page object being a key.

That seems stupid though. There has to be a better solution here.

And just so I don't leave anything out, here is my html javascript code for parsing the JSON (I'm using the yahoo user interface library). I got this code from the YUI site and barely changed it at all:

code:

YAHOO.util.Connect.asyncRequest('GET',"/getPages?myAdventureKey={{ adventure.key }}", callbacks);

#this is the callback function, since the request was async
success : function (o) {
  var messages = [];
  try {
    messages = YAHOO.lang.JSON.parse(o.responseText);
  }
  // The returned data was parsed into an array of objects.
  // Add a P element for each received message
  for (var i = 0, len = messages.length; i < len; ++i) {
    var m = messages[i];
    //m.name is the thing we want
    //i'm also using the tree view class from YUI
    var tmpNode = new YAHOO.widget.TextNode(m.name, tree.getRoot(), false);
node, false);
  }

Here is the documentation for the YUI JSON stuff: http://developer.yahoo.com/yui/examples/json/json_connect.html

# ? Dec 6, 2008 04:40

tayl0r: Oct 3, 2002; This post brought to you by SOE

my post was probably too long to fit in this thread.. i just made a new one: http://forums.somethingawful.com/showthread.php?threadid=3026496

# ? Dec 6, 2008 04:52

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

tayl0r posted:

I get back this JSON:
code:
{"6": {"adventure": 5, "name": "page name", "id": 6}}
Notice there are no enclosing square brackets, which you need for valid JSON (I think). This JSON does not read into my javascript at all, so I'm assuming its not valid JSON.

That is valid JSON. Your javascript code must be incorrect.

tayl0r posted:

I manually tested this JSON string, which does get parsed into my java script:
code:
[
  {"adventure": 5, "name": "page name", "id": 6}
]
So, here is my question: how the gently caress can I make this work?
* I need to manually enclose the entire JSON string with square brackets.
* I need to change my pagesJSON object to be more like an array instead of a dict/hash, so it just has a raw list of page objects inside it, instead of each page object being a key.

That seems stupid though. There has to be a better solution here.

If you want a list instead of a mapping, have you considered serializing a list?

code:

simplejson.dump([p.toDict() for p in pages], self.response.out)

# ? Dec 6, 2008 04:53

tayl0r: Oct 3, 2002; This post brought to you by SOE

Ok, good to know that is valid json. The javascript example code I got from YUI probably just doesn't work with that format.

I actually already did what you suggested (use a list instead of a dictionary) and it generates the json that works with my javascript. Although, my code is 4 lines and yours is just 1 =P

Thanks for tip, Janin.

# ? Dec 6, 2008 05:19

inveratulo: May 14, 2002; wat

I have been stumped trying to get this one piece of code to function properly. I am trying to use the pxssh library to execute commands on a remote shell. Whenever I try to put the sendline() commands in a for loop, it will run the first command, but just hang on the subsequent commands and never execute. I cannot for the life of me figure out why this is happening.

Can someone shed some light on this?

This works:

code:

conn = pxssh.pxssh()
... login stuff is going here
...
conn.sendline("ls -latr")
conn.prompt()
print conn.before
conn.sendline("hostname -a")
conn.prompt()
print conn.before

but this does not work:

code:

conn = pxssh.pxssh()
cmds = open("commandFile.txt",'r')
... login stuff is going here
...
for line in cmds:
  conn.sendline(line.strip())
  conn.prompt()
  print conn.before

# ? Dec 8, 2008 21:53

king_kilr: May 25, 2007

I'm looking for a canvas library. Basically I'd like to emulate the HTML canvas element, specifically I need to be able to add polygons, whose color has an alpha channel/transparency, I need to be able to edit polygons on the canvas, and I need to be able to get the actual color for a given pixel. Basically I'm trying to implement http://alteredqualia.com/visualization/evolve/ in python

# ? Dec 12, 2008 02:23

tef: May 30, 2004; -> some l-system crap ->

Generate SVG if you want vector output, but you may find that the python imagine library does enough for your needs.

# ? Dec 12, 2008 04:51

tripwire: Nov 19, 2004; _{ghost flow}

I just figured out the coolest thing (its probably not new to lot of you)

If you are on linux and you can write to /dev/dsp and hear sound, check this out :smug:

code:

import math    
try:
    audio = open('/dev/dsp','w')
except IOError:
    exit()

for x in xrange(1,100000):
    a =  chr (int ( max( min (128 * math.sin((x ** math.log(x**0.125)) * math.pi/180),255),0 ) ) )
    audio.write('%c' % a)
audio.flush()
audio.close()

Edit: whoa, try cranking the xrange value up and using this for a instead:
chr (int ( max( min (128 * math.sin((x ** math.log( (x//5 % 23 + 1) **0.2)) * math.pi/180), 255),0) ) )
:2bong:

tripwire fucked around with this message at 06:34 on Dec 12, 2008

# ? Dec 12, 2008 06:10

king_kilr: May 25, 2007

I'm super excited to say that my PyCon talk was accepted! It's a panel on ORM philosophies and design decisions with Jacob Kaplan-Moss, Ian Bicking, Mike Bayer, Guido van Rossum, and Massimo Di Pierro. I know we had a few other goons who submitted talk proposals, what were your results?

# ? Dec 16, 2008 02:09

tripwire: Nov 19, 2004; _{ghost flow}

king_kilr posted:

I'm super excited to say that my PyCon talk was accepted! It's a panel on ORM philosophies and design decisions with Jacob Kaplan-Moss, Ian Bicking, Mike Bayer, Guido van Rossum, and Massimo Di Pierro. I know we had a few other goons who submitted talk proposals, what were your results?

Thats awesome, congratulations

# ? Dec 16, 2008 03:36

m0nk3yz: Mar 13, 2002; Behold the power of cheese!

king_kilr posted:

I'm super excited to say that my PyCon talk was accepted! It's a panel on ORM philosophies and design decisions with Jacob Kaplan-Moss, Ian Bicking, Mike Bayer, Guido van Rossum, and Massimo Di Pierro. I know we had a few other goons who submitted talk proposals, what were your results?

Both of mine - an intro to multiprocessing, and the "state of concurrency and distributed systems w/ python" were accepted.

# ? Dec 16, 2008 04:38

king_kilr: May 25, 2007

m0nk3yz posted:

Both of mine - an intro to multiprocessing, and the "state of concurrency and distributed systems w/ python" were accepted.

awesome, both of those sound super interesting, can't wait!

# ? Dec 16, 2008 06:39

Capnbigboobies: Dec 2, 2004

Hello all I am a noob to python and programming and I am making an ultra simple script to download files off the net. I got that part working, but for the life of me I cant wrap my head around how to make a progress menu to show how much of the file is left to download. I am using urlgrabber to download the file and it appears that it has some method for reporting how much of the file is left to download but I have no idea how to use it.

# ? Dec 16, 2008 08:56

Kire: Aug 25, 2006

To display a progress bar, I think Tkinter has that capability. But I'm more of a newb than you are so I can't offer anything specific.

# ? Dec 16, 2008 21:27

Capnbigboobies: Dec 2, 2004

Kire posted:

To display a progress bar, I think Tkinter has that capability. But I'm more of a newb than you are so I can't offer anything specific.

This is command line only, gui programming is beyond me.

# ? Dec 17, 2008 14:41

chemosh6969

Jul 3, 2004

code:
cat /dev/null > /etc/professionalism

_{_{I am in fact a massive asswagon.

Do not let me touch computer.}}

Capnbigboobies posted:

This is command line only, gui programming is beyond me.

Just saying, but Tkinter has a download bar as one of the examples that comes with it. You could either use that or at least the code as the basis for a command line one.

edit: For those interested, I found a Python module for sending keystrokes to an active window. It's like AutoIt.

http://www.rutherfurd.net/python/sendkeys/

chemosh6969 fucked around with this message at 19:05 on Dec 17, 2008

# ? Dec 17, 2008 19:03

Kilometers Davis: Jul 9, 2007; They begin again

I literally just (an hour ago) jumped into programming with Python. Right now i'm using this and am up to the 5th part. I'm understanding it so far but I'm not holding much in my memory. Is there a better online tutorial or even book I should use or is this just normal confusion that goes away after using Python for a few weeks.

# ? Dec 18, 2008 04:33

Adbot: ADBOT LOVES YOU

# ? May 24, 2024 01:27

king_kilr: May 25, 2007

IntoTheNihil posted:

I literally just (an hour ago) jumped into programming with Python. Right now i'm using this and am up to the 5th part. I'm understanding it so far but I'm not holding much in my memory. Is there a better online tutorial or even book I should use or is this just normal confusion that goes away after using Python for a few weeks.

If you have some programming background dive into python is pretty good, otherwise I like the regular python tutorial.

# ? Dec 18, 2008 04:55

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »