Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Blotto Skorzany: Nov 7, 2008; He's a PSoC, loose and runnin'
came the whisper from each lip
And he's here to do some business with
the bad ADC on his chip
bad ADC on his chiiiiip

No. AMD64 in common parlance is just a synonym for x86-64. Use stuff built for it if you're on an x64 system, regardless whether Intel or AMD made your processor.

# ? Mar 22, 2010 05:22

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 20:02

yippee cahier: Mar 28, 2005

The only problem is some 3rd party modules may not have x64 binaries built, but hey, using Py3k is going to be the limit in that respect.

# ? Mar 22, 2010 06:07

spankweasel: Jan 4, 2006

OK, I'm running out of places to turn for help on this one.

I have attempted to extend unittest.py to add a subclass of TextTestRunning which takes advantage of the multiprocessing module to run tests in parallel. I have everything working, with the exception of one thing:
(the RuntimeError exceptions - the test suite failures are me testing failures and errors)

code:

> ./run.py 
FEException RuntimeError: 'maximum recursion depth exceeded in __subclasscheck__' in <type 'exceptions.AttributeError'> ignored
Exception RuntimeError: 'maximum recursion depth exceeded in __subclasscheck__' in <type 'exceptions.AttributeError'> ignored
.Exception RuntimeError: 'maximum recursion depth exceeded in __subclasscheck__' in <type 'exceptions.AttributeError'> ignored
.Exception RuntimeError: 'maximum recursion depth exceeded in __subclasscheck__' in <type 'exceptions.AttributeError'> ignored
.Exception RuntimeError: 'maximum recursion depth exceeded in __subclasscheck__' in <type 'exceptions.AttributeError'> ignored

======================================================================
ERROR: test_5 (__main__.T)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./ver3.py", line 104, in test_5
    print foo
NameError: global name 'foo' is not defined

======================================================================
FAIL: test_4 (__main__.T)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./ver3.py", line 101, in test_4
    self.fail ("fail testing")
AssertionError: fail testing

Run Time: 3.022s

I found issue 5508 on this at bugs.python.org but I don't understand how to circumvent these (seemingly) nuisance messages. I also don't understand enough about python internals to really comprehend what's going on in bug 5508. Can somebody help me out here?

code:

#!/usr/bin/python

import sys
import time
import unittest
import Queue
import multiprocessing

class MPRunner(unittest.TextTestRunner):
    def __init__(self, stream=sys.stderr, descriptions=1, verbosity=1):
        super(MPRunner, self).__init__(stream, descriptions, verbosity)
    
    def run(self, test):
        test_queue = multiprocessing.Queue()
        result_queue = multiprocessing.Queue()
        workers = []
        tasks = {}

        result = self._makeResult()
        start = time.time()

        # put each test in the queue
        for tp in test._tests:
            test_queue.put(tp, block=False)
            tasks[tp] = None
        
        # kick off a Process object for each cpu core
        for i in range(multiprocessing.cpu_count()):
            p = multiprocessing.Process(target=runner, 
                                        args=(test_queue, result_queue))
            p.start()
            workers.append(p)

        # process the results
        while tasks:
            try:
                (case, single_result) = result_queue.get(timeout=5)

                # get the errors and failures from the test's result
                result.errors.extend(single_result.errors)
                result.failures.extend(single_result.failures)

                # remove the test from the tasks
                tasks.pop(case)

            except Queue.Empty:
                break

        stop = time.time()
        result.printErrors()
        self.stream.writeln("Run Time: %.3fs" % (stop-start))

        # Tell all the workers to stop
        for w in workers:
            if w.is_alive():
                test_queue.put("STOP", block=False)

        return result

def runner(test_queue, result_queue):
    # simple function to get the next entry in the test_queue
    def get_test():
        return test_queue.get()

    # create a new result object for each single test
    def makeResult():
        stream = unittest._WritelnDecorator(sys.stderr)
        res = unittest._TextTestResult(stream, descriptions=1, verbosity=1)
        return res

    try:
        try:
            for case in iter(get_test, "STOP"):
                try:
                    single_result = makeResult()
                    case(single_result)
                    result_queue.put((case, single_result))
                except KeyboardInterrupt, SystemExit:
                    raise
                except:
                    raise
        except Queue.Empty:
            pass
    finally:
        test_queue.close()
        result_queue.close()

class T(unittest.TestCase):
    def setUp(self):
        pass
    def tearDown(self):
        pass
    def test_1(self):
        time.sleep(1)
    def test_2(self):
        time.sleep(2)
    def test_3(self):
        time.sleep(3)
    def test_4(self):
        # fail
        self.fail ("fail testing")
    def test_5(self):
        # error
        print foo

if __name__ == "__main__":
    s = unittest.TestLoader().loadTestsFromTestCase(T)
    MPRunner().run(s)

# ? Mar 22, 2010 23:00

Sweeper: Nov 29, 2007; The Joe Buck of Posting; Dinosaur Gum

Has anyone done anything with Python and SOAP here? I have a project where I need to connect to a WCF service using Python. I am using suds and it was working great until I tried to use the service. I can create things with the factory and I can grab all of the methods and ports, but whenever I try to run a function from one of the ports it spits back Error 415 (I think, may be 451?). It is something about expecting application/soap+xml but got text/xml. I think this might be because the server could be running SOAP 1.1 and not 1.2, but I am not sure. Is there any way to force suds to expect text/xml? Should I be using something other than suds?

# ? Mar 23, 2010 00:05

Scaevolus: Apr 16, 2007

spankweasel posted:

code:

> ./run.py 
FEException RuntimeError: 'maximum recursion depth exceeded in __subclasscheck__' in <type 'exceptions.AttributeError'> ignored

Try

code:

import sys

sys.setrecursionlimit(1500)

# ? Mar 23, 2010 02:29

spankweasel: Jan 4, 2006

Scaevolus posted:

Try

code:

import sys

sys.setrecursionlimit(1500)

Well, something new!

Now I know -where- the offending __getattr__ is...

code:

> ./go.py 
.Traceback (most recent call last):
  File "./go.py", line 110, in <module>
    MPRunner().run(s)
  File "./go.py", line 39, in run
    (case, single_result) = result_queue.get(timeout=5)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/queues.py", line 105, in get
    res = self._recv()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 662, in __getattr__
    return getattr(self.stream,attr)

< that same getattr error times 300 entries>

RuntimeError: maximum recursion depth exceeded

code:

unittest.py

class _WritelnDecorator:
    """Used to decorate file-like objects with a handy 'writeln' method"""
    def __init__(self,stream):
        self.stream = stream

    def __getattr__(self, attr):
        return getattr(self.stream,attr)

    def writeln(self, arg=None):
        if arg: self.write(arg)
        self.write('\n') # text-mode streams translate to \r\n if needed

I guess I'll try to write my own version of _WritelnDecorator which doesn't use __getattr__

Thanks for at least a breadcrumb on where the hell to look. Man, that issue is -EVERYWHERE- in anything using python2.6. Why the hell was the bug closed as "Nope, nothing to see here." with this many people suffering from this issue.

# ? Mar 23, 2010 03:09

Sock on a Fish: Jul 17, 2004; What if that thing I said?

Sweeper posted:

Has anyone done anything with Python and SOAP here? I have a project where I need to connect to a WCF service using Python. I am using suds and it was working great until I tried to use the service. I can create things with the factory and I can grab all of the methods and ports, but whenever I try to run a function from one of the ports it spits back Error 415 (I think, may be 451?). It is something about expecting application/soap+xml but got text/xml. I think this might be because the server could be running SOAP 1.1 and not 1.2, but I am not sure. Is there any way to force suds to expect text/xml? Should I be using something other than suds?

I've never had good experience consuming a WSDL with any utilities available in Python. I always end up writing a library to implement the calls I need.

I think PyWebSvcs must only get tested against something available in Zope.

# ? Mar 23, 2010 06:33

ATLbeer: Sep 26, 2004; Über nerd

If anyone is interested in NoSQL stuff I'm writing a series on Redis, with Python as the main development language. Have 2 articles done right now.

http://degizmo.com/series-redis-tutorials/

# ? Mar 23, 2010 19:46

DearSirXNORMadam: Aug 1, 2009

How do I get readline()/realines() to actually talk to specific lines in the file, rather than just the first line and all the lines up to the size limit respectively?

Ie, I have an EMBL file, it contains a lot of poo poo, I need to sort through line by line until I find '//'.

I guess I could just do it by doing

code:

bob = open('/whatever/data.dat','r')

hugeList = bob.readlines()

while hugeList[i]!='//':
  i++

""i is now the index of '//'""

But this seems hugely wasteful if I just have to talk to the first 200 lines of a 15kb file.

Thus, how do I tell readline() to search through the line until it passes 600 lines, then return line #601?

Disclaimer: I'm a noob, if I'm missing something stupid, it's probably because I'm ignorant. Make sure to point it out.

# ? Mar 23, 2010 23:22

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

A file is just a sequence of characters. readline has no way of knowing where a newline is or where a particular string is without reading through the file in order.

# ? Mar 23, 2010 23:52

Sweeper: Nov 29, 2007; The Joe Buck of Posting; Dinosaur Gum

Sock on a Fish posted:

I've never had good experience consuming a WSDL with any utilities available in Python. I always end up writing a library to implement the calls I need.

I think PyWebSvcs must only get tested against something available in Zope.

I got suds working, I had to edit the bindings.py because it was hard coded to one particular namespace and I needed it to be another namespace. Now the only problem I have it that it needs to use the integrated windows authentication stuff... under Linux... I need to use gssapi from what I have seen.

# ? Mar 24, 2010 00:38

Scaevolus: Apr 16, 2007

Mirconium posted:

But this seems hugely wasteful if I just have to talk to the first 200 lines of a 15kb file.

It's 2010; 15KB is nothing. This is an especially silly optimization to attempt in a language like Python.

code:

ebml_file = open('/whatever/data.dat')

for line in ebml_file:
    if line.strip() == '//':
        break

line = ebml_file.readline()

e: vvv

Scaevolus fucked around with this message at 04:20 on Mar 24, 2010

# ? Mar 24, 2010 01:35

tripwire: Nov 19, 2004; _{ghost flow}

Scaevolus posted:

It's 2010. 15KB is nothing. This is an especially stupid "optimization" to attempt in a language like Python.
code:
ebml_file = open('/whatever/data.dat')

for line in ebml_file:
    if line == '//':
        break

line = ebml_file.readline()

Make sure you strip the trailing newline or that test for equality with '//' will never evaluate true!

# ? Mar 24, 2010 01:42

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

Does anyone know a lot about Python buffer objects? I want to do something like NumPy and take an optional argument for (C extension) functions that will provide a buffer to use to fill a C array that gets put into a NumPy array. Basically:

code:

my_numpy_array = my_function(butts, out=my_buffer)

The NumPy documentation isn't exactly clear on how it deals with buffer objects, and it's not even very easy to figure out when the new buffer interface got added to Python.

# ? Mar 24, 2010 19:56

tripwire: Nov 19, 2004; _{ghost flow}

Avenging Dentist posted:

Does anyone know a lot about Python buffer objects? I want to do something like NumPy and take an optional argument for (C extension) functions that will provide a buffer to use to fill a C array that gets put into a NumPy array. Basically:
code:
my_numpy_array = my_function(butts, out=my_buffer)
The NumPy documentation isn't exactly clear on how it deals with buffer objects, and it's not even very easy to figure out when the new buffer interface got added to Python.

I'm not sure if this is entirely relevant, but I was reading Cython docs a while back and it mentioned that efficient indexing into arrays is supported by the buffer object, but only (natively) in python 3; in python 2 theres some hackery going on in cython to emulate the same thing. Again, I'm not sure if this even has anything to do with what you are asking, but skim through this and see if anything jumps out at you: http://wiki.cython.org/enhancements/buffer

# ? Mar 24, 2010 20:10

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

Yeah, that's not really what I need. I just want to know how to take an object from Python into C, get its buffer interface, and then use that as a block of (contiguous) memory to pass to an internal C function, and then finally put that block into a NumPy array. If NumPy's source weren't such a bitch to read, I'd just look through it to see how they do it.

I mean, I could probably figure it out, but this seems like an area where memory issues are rife, and I wanted to see what other people's experiences were (if any).

Avenging Dentist fucked around with this message at 20:34 on Mar 24, 2010

# ? Mar 24, 2010 20:32

BigRedDot: Mar 6, 2008

The buffer interface is pretty new, they were still working on getting it added when I was at Enthought a few years ago, unless I am misremembering. Have you tried just asking the numpy mailing list? Travis is usually pretty good about answering technical questions.

# ? Mar 24, 2010 20:37

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

It looks like NumPy only uses the new buffer protocol in 1.5 (and that the buffer protocol is a 2.6+ thing). I think I'll wait until NumPy 1.5 is out before worrying about this overly much.

# ? Mar 24, 2010 23:08

UberJumper: May 20, 2007; woop

Is there any decent library for basically doing 2d geometry? Aside from shapely? I keep running into tiresome issues with checking to see if polygons intersect, using a line to splice a polygon, etc.

# ? Mar 25, 2010 22:22

Stabby McDamage: Dec 11, 2005; Doctor Rope

What is the canonical way to read a file one byte at a time?

Because python doesn't allow assignment in the conditional, I've been doing something like this:

code:

while True:
	c = f.read(1)
	if not c: break
	...

Because that's ugly, I made a generator to hide it:

code:

def bytes(f):
	while True:
		c = f.read(1)
		if c: yield c
		else: return

for c in bytes(f):
	...

Is there a prettier way to do that?

More generally, is there an idiom that means "keep giving me the return value of this expression until it returns false"? Something analagous to C's "while (c=func()) {}"? It seems to come up in various situtations, and I've been using while True with conditional breaks.

# ? Mar 26, 2010 20:44

king_kilr: May 25, 2007

I'd be lazy and do

[char for line in file for char in line]

But I'm really lazy

# ? Mar 26, 2010 20:53

nbv4: Aug 21, 2002; by Duchess Gummybuns

Stabby McDamage posted:

More generally, is there an idiom that means "keep giving me the return value of this expression until it returns false"? Something analagous to C's "while (c=func()) {}"? It seems to come up in various situtations, and I've been using while True with conditional breaks.

The idiom is to make that function a generator. Any function that "keeps giving me the return value of this expression until it returns false", is basically what a generator is for.

edit: you can also raise StopIteration in generators which will do the same thing as 'return False' if you don't like conditional breaks

nbv4 fucked around with this message at 21:44 on Mar 26, 2010

# ? Mar 26, 2010 21:37

AbsentMindedWelder: Mar 26, 2003; It must be the fumes.

I have never posted in SH/SC or TCOC until now, so allow me to introduce myself. I started loving with computers around '96, and went into a professional IT career (which is now deceased), but never programmed anything except QBASIC, HTML, or basic Linux scripts. I've been a Linux geek for a while, and after 14 years of PC's, am finally starting to teach myself something more interesting.

So, on to my question...

After having have only worked with python for about 2 days, I decided it would be fun to hook up a breadboard with 7 segment displays and decoders to my parallel port. I have a function that can take a decimal or hex value and do what I want with it.

The range of values I want to send to this function are from 0x00 - 0x99.

My problem is the output of the function which calls the I/O function is producing a string such as '0x99' that the I/O function won't accept because it isn't a real numerical value.

So my REAL question is...

How do I change '0x99' to 0x99?

AbsentMindedWelder fucked around with this message at 22:26 on Mar 26, 2010

# ? Mar 26, 2010 22:24

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

int('0x99', 16)

# ? Mar 26, 2010 22:32

Stabby McDamage: Dec 11, 2005; Doctor Rope

nbv4 posted:

The idiom is to make that function a generator. Any function that "keeps giving me the return value of this expression until it returns false", is basically what a generator is for.

edit: you can also raise StopIteration in generators which will do the same thing as 'return False' if you don't like conditional breaks

Okay then -- can the generator I wrote be improved?

Basically, I'm just not used to using infinite loops with breaks for doing something as simple as byte-wise reading. If that's really the best way, then I'm a bit surprised.

# ? Mar 27, 2010 00:23

Allie: Jan 17, 2004

Yes, while True: is a common idiom. Sorry to disappoint.

Also, I'm not sure I'd create a generator just to hide that if you're only doing this in one place.

# ? Mar 27, 2010 00:28

AbsentMindedWelder: Mar 26, 2003; It must be the fumes.

Avenging Dentist posted:

int('0x99', 16)

Thanks, worked perfectly! :cheers:

Here is my first official python program. Please critique.

The hardware hooked up to the parallel port is basically just (2) 7447 IC's hooked up to (2) 7 segment displays, and a few resistors. I'm using pyParallel to handle the I/O functions. (This is an old machine running Debian with no GUI)

code:

#!/usr/bin/python
# Filename: count.py

import time
import parallel
p = parallel.Parallel()
p.setData(0)

def ioprint(num):
	value = '0x' + str(num)
	print(num)
	p.setData(int(value,16))

def countup():
	num = -1
	up = True
	while up:
		num = num + 1
		ioprint(num)
		time.sleep(.25)
		if num == 99:
			up = False

def countdown():
	num = 100
	down = True
	while down:
		num = num - 1
		ioprint(num)
		time.sleep(.25)
		if num == 0:
			down = False

loop = True
while loop:
	countup()
	time.sleep(2)
	countdown()
	time.sleep(2)

If you decide to use pyParallel you'll probably have to do 2 things to get it to work.

-Edit setup.py to make it install correctly
-rmmod lp

If anyone is curious enough for a circuit diagram, you can pretty much figure it by looking up the 7447 documentation and a pin-out of a standard parallel port. Also this is a great website about interfacing the parallel port.

# ? Mar 27, 2010 01:06

nbv4: Aug 21, 2002; by Duchess Gummybuns

Milde posted:

Yes, while True: is a common idiom. Sorry to disappoint.

Also, I'm not sure I'd create a generator just to hide that if you're only doing this in one place.

Not in the case of reading lines from a file. IMO the most pythonic way is what king_kilr said, for line in file and then for char in line

# ? Mar 27, 2010 01:10

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

nbv4 posted:

Not in the case of reading lines from a file. IMO the most pythonic way is what king_kilr said, for line in file and then for char in line

Why would you want to double-buffer your input?

# ? Mar 27, 2010 01:15

nbv4: Aug 21, 2002; by Duchess Gummybuns

Avenging Dentist posted:

Why would you want to double-buffer your input?

i didn't say it as the most efficient way...

Anyways, I thought file objects were smart enough to be iterated over without needing to be buffered?

# ? Mar 27, 2010 01:43

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

nbv4 posted:

Anyways, I thought file objects were smart enough to be iterated over without needing to be buffered?

I'm talking about the internal buffer at the C level (Python file objects ultimately forward on to FILE*s in C) plus the additional "buffer" of the current line. This is further complicated by the fact that you have to look ahead to find where the current line ends, unlike with fixed-size buffers where you know the buffer size a priori.

File I/O is one of the few places I'd be concerned about (non-asymptotic) efficiency in Python, since unnecessary reads from disk are slow even in Python terms.

# ? Mar 27, 2010 01:54

Scaevolus: Apr 16, 2007

dv6speed posted:

Thanks, worked perfectly!

Here is my first official python program. Please critique.

If you haven't already done so, you should skim over the Python Tutorial. It's pretty short, and introduces you to syntax that you might not run into otherwise.

Set your editor to indent with spaces instead of tabs.

Suggestions:

code:

#!/usr/bin/python
# Filename: count.py

import time
import parallel

p = parallel.Parallel()
p.setData(0)

def ioprint(num):
    print num
    p.setData(int(str(num), 16))

def countup():
    for num in xrange(0, 100):
        ioprint(num)
        time.sleep(.25)

def countdown():
    for num in xrange(99, -1, -1):
        ioprint(num)
        time.sleep(.25)

while True:
    countup()
    time.sleep(2)
    countdown()
    time.sleep(2)

Are you using Python 3? If not, you shouldn't have parentheses around print statements. It's not a function in Python 2.x.

Scaevolus fucked around with this message at 02:25 on Mar 27, 2010

# ? Mar 27, 2010 02:16

Scaevolus: Apr 16, 2007

Avenging Dentist posted:

File I/O is one of the few places I'd be concerned about (non-asymptotic) efficiency in Python, since unnecessary reads from disk are slow even in Python terms.

Is there a modern OS that doesn't cache page reads?

# ? Mar 27, 2010 02:28

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

Scaevolus posted:

Is there a modern OS that doesn't cache page reads?

Paging doesn't have anything to do with this, since that would happen after reading from disk into a buffer.

But then, I don't actually know (or care) how Python implements readline() so it's possible that if a line would be split by the file buffer that it won't go out to disk more than necessary.

I suppose it's a more immediate concern that you'd have to figure out how much to read from the buffer every time you call readline(), which is unnecessary.

# ? Mar 27, 2010 02:47

AbsentMindedWelder: Mar 26, 2003; It must be the fumes.

Scaevolus posted:

Are you using Python 3? If not, you shouldn't have parentheses around print statements. It's not a function in Python 2.x.

I am reading "A byte of Python" for versoin 3.x, so yes.

# ? Mar 27, 2010 02:51

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

dv6speed posted:

I am reading "A byte of Python" for versoin 3.x, so yes.

You really shouldn't use Python 3 unless you have a good reason why you need to. It's just going to bite you in the rear end in the future.

# ? Mar 27, 2010 02:54

AbsentMindedWelder: Mar 26, 2003; It must be the fumes.

Avenging Dentist posted:

You really shouldn't use Python 3 unless you have a good reason why you need to. It's just going to bite you in the rear end in the future.

Why do you say that?

# ? Mar 27, 2010 02:59

good jovi: Dec 11, 2000

dv6speed posted:

Why do you say that?

Mostly because very few modules have been ported over to it, and if you ask people for help, odds are that they're going to be more familiar with python 2. If you feel really strongly about leading the way into the future or something like that, by all means. Its deficiencies are not technical, but social.

# ? Mar 27, 2010 03:03

Avenging Dentist: Oct 1, 2005; oh my god is that a circular saw that does not go in my mouth aaaaagh

dv6speed posted:

Why do you say that?

There are still an awful lot of important third-party Python modules that don't have Python 3 support yet, and chances are you're going to decide you want one of them and then have to either 1) go back to Python 2 and unlearn a bunch of stuff, or 2) wait 6 months to a year until they release a Python 3 version.

# ? Mar 27, 2010 03:04

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 20:02

Stabby McDamage: Dec 11, 2005; Doctor Rope

Avenging Dentist posted:

Paging doesn't have anything to do with this, since that would happen after reading from disk into a buffer.

But then, I don't actually know (or care) how Python implements readline() so it's possible that if a line would be split by the file buffer that it won't go out to disk more than necessary.

I suppose it's a more immediate concern that you'd have to figure out how much to read from the buffer every time you call readline(), which is unnecessary.

My most immediate concern is that it's a binary file that may go for an arbitrary length with no newlines, so reading line-by-line would mean it has a potentially unbounded memory requirement.

# ? Mar 27, 2010 03:37

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »