Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Mashi posted:

Can someone explain what pypy is and how it all fits together? I have looked at the website and I don't quite understand it, you have python written in python, then RPython, then something which hopefully will one day be faster than C. How does it work?

I am by NO means a PyPy expert, though I do think it's pretty cool.

The developers say PyPy is a tool for generating virtual machines. PyPy started as a Python interpreter written in Python, and it can still be used as such on top of CPython and you get an ungody slow Python interpreter. And so, they decided to write a translation layer that takes RPython (a subset of Python) and translates it into C code. They then wrote the interpreter in RPython and now the interpreter can be translated into C code, and is currently around twice as slow as CPython.

However, they can use a different translator to automatically translate PyPy onto different platforms such as CLI/.NET, JVM, and apparently also less obviously useful platforms such as Smalltalk and Scheme.

Of course, the translator will translate anything from RPython to C/CLI/JVM or whatever, so people can write whatever interpreters or other programs they like in RPython (though using RPython in production isn't really advised right now, from what I can tell). You can also use RPython to generate libraries for CPython that have been translated into C and compiled.

Anyway, they're working on things like JIT compilation/optimization and messing around with writing plugins for PyPy to change the behavior of the language in various ways (like reimplementing Stackless).

Really, the interesting thing to me is that if PyPy really takes off, it promises to make Jython and IronPython and CPython obsolete, since it could be run on pretty much any platform. I'd also like to see a stable RPython that would be useful for application developers to write high performance Python. Of course, if they get the JIT compiler in order, you should pretty much get the same speed boost out of any piece of your code that conforms to the constraints of RPython, even if it's embedded in the middle of your Python program. Sadly, I keep reading that the JIT is currently pretty seriously broken, so who knows if it'll ever pan out.

If you're interested in the gritty details of it all, this Google Tech Talk goes into some depth on it. The sound gets better after a little while, but it's still kinda a lacking presentation.

# ¿ May 22, 2008 01:41

Adbot: ADBOT LOVES YOU

# ¿ May 20, 2024 04:38

Benji the Blade: Jun 22, 2004; Plate of shrimp.

bitprophet posted:

You know, trying to document what it means to be "Pythonic", sorta like that site I probably have bookmarked but can't find right now (guy has a nice big list of Python idioms and what to do / not do).

I think this is what you're referring to: http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html. It's an excellent guide for people who've got the Python syntax down but aren't quite thinking in Python yet.

The Google Python standards are generally pretty good, but contradicting PEP 8 by suggesting two spaces instead of four is, in my mind, a terrible idea. It may work well inside Google considering they have a whole lot of in-house code, but when you're programming by yourself or in a small team, this is just going to cause a lot of headaches. The problem is not stylistic, like it would be in C or C++, but that using the wrong number of spaces might cause subtle incorrect behavior if someone forgets or if someone's editor is set up wrong, and to me, the benefit is pretty questionable.

# ¿ Sep 8, 2008 17:35

Benji the Blade: Jun 22, 2004; Plate of shrimp.

MeramJert posted:

code:

import math, wave, array, stackless

class Sine:
         
    def __init__(self, seconds, frequency, amplitude, channel, sampleRate=44100):
        [...]
        self.data = sampleArray        
        stackless.tasklet(self.send_array)(channel)
        
    def send_array(self, channel):
        channel.send(self.data)

code:

import pyaudio, stackless

class OutStream:
    
    def __init__(self, Format=pyaudio.paInt16, Channels=1, sampleRate=44100):
        self.channel = stackless.channel()
        self.p = pyaudio.PyAudio()
        self.stream = self.p.open([...])
        stackless.tasklet(self.listen)()
        
    def listen(self):
        while 1:
            data = self.channel.receive()
            stackless.tasklet(self.play)(data)
        
    def play(self, data):
        self.stream.write(data)
        
    def __del__(self):
        self.stream.close()
        self.p.terminate()

code:

import stackless
from Sine import *
from Output import *

output = OutStream()
output2 = OutStream()

x = Sine(1,650,75,channel=output.channel)
x = Sine(1,450,75,channel=output2.channel)

stackless.run()

Tripwire addressed the correct way to handle this from a sound perspective, but from a Stackless perspective, there's no reason your code should run concurrently. Remember that Stackless is cooperative multitasking, so in order for one tasklet to continue execution, another must give it up, either by sleeping (and letting the scheduler move on to the next tasklet) or else by blocking on a channel.

In your case, you have the Sine class send a single chunk of data over a channel to OutStream.listen, which spawns a tasklet of OutStream.play, which in turn uses a traditional blocking call (if I'm not mistaken) to write to the audio device. Once it's done, OutStream.play completes and only NOW does the Stackless scheduler kick in. And once it comes back around to the OutStream.listen that spawned the play tasklet, it'll block forever on its channel, since the sign only ever writes to that channel once.

Try instead making Sine.send_array send chunks of the data interspersed with calls to stackless.schedule() to make sure other tasklets are getting scheduled.

Basically, remember that control doesn't pass between tasklets unless you TELL it to (again, by blocking on a read to or write from a channel, sleeping, or calling stackless.schedule directly).

# ¿ Feb 6, 2009 07:03

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Kaluza-Klein posted:

Ha, I just came here to edit my post with this:

code:

	dirList = []
	for item in fileList:
		dirList.append(item[0])

	for item in fileList:
		num = dirList.count(item[0])
		print "%s appears %s time(s)" % (item[0], num)

For what it's worth, here's what I came up with:

code:

def f(file_list):
    from itertools import groupby
    for d, f_itr in groupby(sorted(file_list), lambda item: item[0]):
        files = [item[1] for item in f_itr]
        if len(files) == 1:
            pass # do stuff
        else:
            print "%s appears %i times" % (d, len(files))

This has the benefit of having better performance for large file_lists (O(n log(n)) instead of O(n^2)), but if it's just gonna be a small number, my gut tells me the shorter version will run faster.

Mostly, I wanted to introduce the idea of using "sorted" and "groupby" to solve your problem. In this case, they're a lot like using the UNIX utils, "sort | uniq" only instead of just deleting multiples, groupby gives you all the values in the list where the key function (in this case "lambda item: item[0]") evaluates to be equal.

# ¿ Feb 25, 2009 17:31

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Edit:: ^^ This ^^

numData is almost certainly a reference to the same list in every item in ranges. You don't include the code where you assign numData, but make sure you're constructing a new object for each entry in ranges if you want each numData to be different.

Again, variables in Python are just labels that point to data somewhere in memory. The equals operator doesn't copy anything but references (at least usually).

# ¿ Feb 27, 2009 22:30

Benji the Blade: Jun 22, 2004; Plate of shrimp.

such a nice boy posted:

ljust returns a new string.

tag = tag.ljust(len(tag) + 1 + len(tag)%2, '\0')

That won't do anything, because you assign the left justified returned string to tag, which is then overwritten at the top of the for loop by whatever's next in the list.

Instead, you'll want to build a new list:

code:

def adjust_strs(str_list):
    return [tag.ljust(len(tag) + 1 + len(tag) % 2, '\0') for tag in str_list]

adjusted_strs = adjust_strs([make, model, software, time, uniquecameramodel, copyright])

If it's really important to do it in place, you could do this instead:

code:

def adjust_strs_inplace(str_list):
    for i in range(len(str_list)):
        str_list[i] = str_list[i].ljust(len(tag) + 1 + len(tag) % 2, '\0')

adjusted_strs = [make, model, software, time, uniquecameramodel, copyright]
adjust_strs_inplace(adjusted_strs)

But that's a little more verbose and may not actually be any faster, depending on the data. In this case, you're still not altering the strings in place, you're just not creating a new list, but instead replacing all the items in your first list.

Benji the Blade fucked around with this message at 19:10 on Feb 28, 2009

# ¿ Feb 28, 2009 19:05

Benji the Blade: Jun 22, 2004; Plate of shrimp.

supster posted:

My goal here is to override the .x attribute of Parent in Child with a property.

http://pastie.org/438066

The error message leads me to believe that super(Child, self) is not returning an instance of the Parent class (or a proxy for the Parent class or whatever) as I am expecting it to.

There are a couple problems here.

First, when I run similar code to yours using 2.5, I get "TypeError: 'super' object has only read-only attributes (assign to .x)" in Child's x setter. I don't know a ton about objects of type super, but it appears you can't assign to one.

Second, you have a more basic misunderstanding when you write "super(Child, self).x = self._x + 1". This is supposing that there is Parent's x attribute hiding on your Child object behind its x property, but that's not the case. The x property overwrites the 'x' slot on that object. Notice that we can still access a parent's methods, but that's because we look them up from the parent class' namespace, not the self object's namespace.

So, even if your x setter worked, it would be recursive, and would terminate when you ran out of stack space.

# ¿ Apr 6, 2009 17:38

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Janin posted:

I'm almost certain that PyGame also uses hardware acceleration, because IIRC it's just a wrapper around SDL. I've used it for OpenGL-based stuff before.

Last I checked, while SDL has tools to provide an OpenGL context, it does not provide any sort of hardware-accelerated support for things such as drawing, rotating, or scaling sprites. For that, you'd have to write your own OpenGL calls.

Things like SDL_BlitSurface provide, IIRC, just your usual software blit.

# ¿ Apr 6, 2009 18:32

Benji the Blade: Jun 22, 2004; Plate of shrimp.

supster posted:

This doesn't really make sense - you are implying that all functions in Python are virtual, but clearly it's working in the constructor. Is the __init__ function a special case and not virtual?

When you write self.foo, it looks "foo" up in the object's __dict__. When you write Parent.foo, it looks "foo" up in the class' __dict__. In this case, __init__ isn't a special case because you're (by way of super) calling the "__init__" from the class' __dict__, not from the object's __dict__. The one in the object's __dict__ is from the last class that defined it, in this case, Child.

You can sort of think of all methods being virtual.

supster posted:

If this is the case, is there specifically a different way I can access the base class's method that has been overriden?

BaseClass.method(self)

supster posted:

If x was a method in the base class I think I could just call Parent.x(self, value) and be fine, but because it is an attribute I can't do that.

Yes, but...

The issue is that Parent.x doesn't exist. If we have an object 'p' of type Parent, then the __init__() method will add attribute x to p ("p.x"), but in this case, it doesn't change anything in the class' namespace, only the object's.

Then, when you make an attribute x on class Child and call Parent.__init__(c) where c is an object of class Child, Parent.__init__ tries to set x on c, but since c is already a property (from the definition of class Child), it tries to call the setter.

I wish I could explain this better. Sorry I'm not doing a terribly good job of it.

# ¿ Apr 6, 2009 21:48

Benji the Blade: Jun 22, 2004; Plate of shrimp.

supster posted:

So are you saying that "__init__" in a class's dict will always be that class's constructor and "__init__" in an object's dict will always be that object's parent class's constructor (if it inherited from one)?

The object's __init__ will start out as the __init__ from the object's class (if that class defined __init__). You can access the parent class' __init__ (and similarly anything in their namespaces) by either using Parent.__init__ or super(Child, c).__init__, which in this case does basically the same thing, but performs some black magic if you're doing multiple inheritance.

supster posted:

Ok, I understand. So there's no internal instance of the parent object in the child object instance? I have no way of accomplishing what I am trying to do?

Well, you could add a separate method or property that did the right thing, but then it wouldn't work with code that called the old property. You could instead use the adapter pattern to wrap the Parent rather than using inheritance, which I think is probably the right thing to do.

There might be a better way to do it, but I'd need to give it some thought.

I'm glad to hear it's been helpful. I promise once you get it, this is all a lot simpler than it sounds.

# ¿ Apr 6, 2009 23:20

Benji the Blade: Jun 22, 2004; Plate of shrimp.

supster posted:

The reason I didn't want to do this was because the Parent class has another dozen or so attributes and methods that are being used, so I didn't want to have to have to change the usage of all of those to Adapater.Parent.method() or have to define all of those methods and attributes in my adapater class.

Python helps you be lazy!

code:

class A(object):
    def f(self): return "base f()"
    def g(self, x): return "base g(%s)" % str(x)

class Adapter(object):
    def __init__(self, base):
        self.base = base

    def f(self): return "adapted f()"

    def __getattr__(self, name):
        return getattr(self.base, name)

And then,

code:

>>> a = A()
>>> adapt = Adapter(a)
>>> adapt.f()
'adapted f()'
>>> adapt.g(12)
'base g(12)'

Now, it doesn't matter how many attributes A has, you only have to override the ones you want, and anything not found in the "adapt" gets called on "adapt.base" via the __getattr__ method.

Edit: Used slightly clearer variable names.

Benji the Blade fucked around with this message at 16:48 on Apr 7, 2009

# ¿ Apr 7, 2009 16:43

Benji the Blade: Jun 22, 2004; Plate of shrimp.

No Safe Word posted:

__getattr__ just let's an object know what to do when something uses the dot operator on it, basically.

foo.bar

could be just as easily expressed:

foo.__getattr__('bar')

Not to be pedantic, but foo.bar would be written foo.__getattribute__('bar'). __getattr__ only gets called for names not in foo.__dict__. *

The names "__getattr__" and "__getattribute__" having different meanings has always rubbed me the wrong way, but regardless, they are useful.

As far as not using names prefixed with __, No Safe Word is spot on, though overriding magic attributes (such as __getattr__) usually indicates there's magic afoot, which is sometimes warranted and sometimes a sign of being overly-clever. However, overriding methods like __init__ and __str__ and __iter__ are totally harmless and very often useful and even required.

* To actually be pedantic, I believe __getattr__ gets called when __getattribute__ throws some exception or other to indicate that it can't resolve name. Usually this is because name isn't in __dict__, but if you override __getattribute__ too, well, god knows what kinda bullshit you've written that may or may not throw an exception.

# ¿ Apr 8, 2009 16:09

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Mustach posted:

not everything is references in Python, either.

I don't understand how this piece of Python shows what you indicate it shows. For the people who didn't follow the link:

quote:

code:
x = 1
y = x
x = 2
print x, y
Output:
code:
2 1

The way I read this, x is set to refer to the immutable object int(1), then y is also set to refer to the immutable object int(1), then x is set to refer to the immutable object int(2). I don't see anything about copy or pass-by-value semantics here. Am I mistaken somehow?

Edit:

supster posted:

Python is also strictly a pass-reference-by-value language, but it's important to understand that immutable objects (e.g., tuple, frozenset) have pass-by-value characteristics because when you make a change to a immutable object a copy is made and reassigned.

How can you make a change to an immutable object? That's a contradiction in terms. You can only come up with a new immutable object and potentially assign it the same name as the old value. If there's "pass-by-value characteristics," I'd like to see 'em. Maybe a code example?

Benji the Blade fucked around with this message at 01:52 on May 15, 2009

# ¿ May 15, 2009 01:48

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Avenging Dentist posted:

This is actually what happens under the hood too. Small ints in the range [-5,256] (I think that's the range) are preallocated by Python and x = 1 just sets x to refer to the global instance of 1 and increments the reference count appropriately.

Yeah. This has the unfortunate side-effect that comparing ints with "is" instead of "==" magically works for small, mostly positive integers, and then your code (usually) breaks on larger numbers or floats. Obviously, you shouldn't be using "is" for that anyway, but it got me when I was new to the language.

Avenging Dentist posted:

This has the amusing side effect that you can change the value of, say, 4 if you use the C API.

Okay, when I said changing an immutable value is a contradiction in terms earlier, that didn't involve breaking the language/runtime. That is hilarious.

# ¿ May 15, 2009 01:56

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Avenging Dentist posted:

code:
x,y = y,x

Or if you need to change the pointed-to lists and not just the names x and y

code:

x[:], y[:] = y[:], x[:]

AD's suggestion is better if just swapping the names does the trick, since it doesn't involve copying.

# ¿ Jul 8, 2009 19:10

Benji the Blade: Jun 22, 2004; Plate of shrimp.

dis astranagant posted:

Plus, there's the can of worms that is making a 2D list (I have a function for that already).

Can of worms?

code:

[[0] * y for _ in range(x)]

# ¿ Jan 22, 2010 18:07

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Sock on a Fish posted:

So it's not recommended to write a dummy web service that does what you think the real one should be doing? That's what I was thinking I'd have to end up doing.

Yes, that's basically what you need to do. Mock up the web service enough to test the client code.

# ¿ May 6, 2010 23:41

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Mr. Clark2 posted:

I throw myself upon the mercy of the python gods...

I know that it can sometimes be difficult to determine intent on the internet, but people are not pressing this issue to make you feel bad or to make fun of you or revel in our superiority. We're pressing you on this to try to figure out what misconception led you to believe that that line would do something so that we can help you learn what's actually going on. There's nothing wrong with not understanding.

# ¿ May 20, 2010 16:37

Benji the Blade: Jun 22, 2004; Plate of shrimp.

It's been a long time since I've worked with XML in any non-trivial way, but it sounds like it'd be pretty trivial to do with XSLT, and I believe lxml can process XSLT. The bonus here is that XSLT might be able to do a decent amount of the processing for you.

The downside is that you'd be writing XSLT code, though. And just to work around a bug in a library in Ubuntu.

# ¿ Jun 2, 2010 20:00

Benji the Blade: Jun 22, 2004; Plate of shrimp.

Two important corrections here:

Stabby McDamage posted:

Short story: CPython's Global Interpreter Lock (GIL) means that only one thread at a time can be executing code.

Only one thread at a time can be executing code inside the Python interpreter. Lots of CPython code spends a lot of its time in C modules that release the lock on the interpreter, like NumPy, I believe. This is not, however, universally true about all C modules. I believe CPython's re module generally doesn't release the GIL.

It still sucks, of course, but significantly less for certain workloads.

Stabby McDamage posted:

Other python implementations, while having slower serial processing performance, do not have this particular limitation.

That's not strictly true either. Stackless and PyPy both, last I heard, have GILs. I believe you're right about Jython and IronPython, though.

# ¿ Jun 17, 2010 19:04

Adbot: ADBOT LOVES YOU

# ¿ May 20, 2024 04:38

Benji the Blade: Jun 22, 2004; Plate of shrimp.

MaberMK posted:

Check out Stackless.

How does Stackless help with a (mostly) CPU-bound problem when Stackless has the same GIL that cpython has?

A non-blocking IO library or multithreading might be able to help utilize the CPU better, but Stackless isn't even really required for that.

# ¿ Sep 14, 2010 21:15

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.