Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

tef: May 30, 2004; -> some l-system crap ->

good jovi posted:

Try upgrading pip

Way ahead of you on this exciting adventure!

code:

$ pip
Traceback (most recent call last):
  File ".../temp/env/bin/pip", line 5, in <module>
    from pkg_resources import load_entry_point

# ? Jun 4, 2014 14:03

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 20:19

tef: May 30, 2004; -> some l-system crap ->

Ooh, I know, how about we update pip, wheel, and setuptools

code:

$ pip install setuptools pip wheel -U
$ pip 
-bash: .../bin/pip: No such file or directory

# ? Jun 4, 2014 14:34

good jovi: Dec 11, 2000

tef posted:

Way ahead of you on this exciting adventure!

ah, huh...

easy_install -U pip?

I've seen similar errors in the past where the pip executable wasn't actually pointed at the same code that was being installed, usually caused by poor $PATH hygiene. The fix there was often running rm `which pip` several times followed by easy_install pip.

# ? Jun 4, 2014 14:34

good jovi: Dec 11, 2000

Are you by any chance using homebrew to install python?

# ? Jun 4, 2014 14:36

tef: May 30, 2004; -> some l-system crap ->

Ok, ok maybe i'll try using easy_install to update pip, setuptools, wheel

It works, although there is a problem with running easy_install now

code:

ImportError: Entry point ('console_scripts', 'easy_install') not found

# ? Jun 4, 2014 14:36

tef: May 30, 2004; -> some l-system crap ->

good jovi posted:

Are you by any chance using homebrew to install python?

This is on an AWS machine.

# ? Jun 4, 2014 14:36

tef: May 30, 2004; -> some l-system crap ->

I think something has either fixed itself or I was trying to debug something on a broken slate. Managed to get everything working again, but honestly I don't believe the workaround is to do 'pip install -U pip wheel' because I am pretty sure I tried this earlier.

# ? Jun 4, 2014 15:33

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Don't use setuptools.

# ? Jun 4, 2014 15:41

Megaman: May 8, 2004; I didn't read the thread BUT...

Suspicious Dish posted:

Don't use setuptools.

Can you elaborate? setuptools is required for lots of things as far as I know, no? What would replace it and why?

# ? Jun 4, 2014 16:09

tef: May 30, 2004; -> some l-system crap ->

Suspicious Dish posted:

Don't use setuptools.

(Well, setuptools being the new version of distribute).

The moral was: if you have to upgrade setuptools and pip, do it one package at a time, especially with old versions of pip.

# ? Jun 4, 2014 16:19

nonathlon: Jul 9, 2004; And yet, somehow, now it's my fault ...

SurgicalOntologist posted:

I didn't get any responses on this the first time around, and this kind of design has come up again and I'm thinking about it.

Isn't this fairly common when you use composition, to "outsource" a protocol to an attribute? I find myself doing this a lot, and while I've never tried to do something tricky like the loops in this quote, I can't help but think there's some O-O concept I'm missing, or if not then there's an opportunity for syntactic sugar.

Having squinted at this for a while, a few thoughts:

* "outsourcing" a protocol to something, sure. Many ways to do this.
* I'm a little unclear however as to whether you're trying to "outsource" for a class or a class instance

I guess your vision is a factory of some sort that you pass a list of "parts" to that will provide functionality for the individual methods?

# ? Jun 4, 2014 16:34

SurgicalOntologist: Jun 17, 2004

outlier posted:

Having squinted at this for a while, a few thoughts:

* "outsourcing" a protocol to something, sure. Many ways to do this.
* I'm a little unclear however as to whether you're trying to "outsource" for a class or a class instance

I guess your vision is a factory of some sort that you pass a list of "parts" to that will provide functionality for the individual methods?

Well I came up with the word delegation, and after googling it turns out that's an actual OOP concept, and I think it's pretty much what I was thinking. Although it's unclear to me how general delegation is supposed to be--maybe I'm just talking one particular sort of delegation.

Yeah, that's why my example wouldn't actually work, because I was mixing levels between class and class instance. I wanted it to be for a class I guess (every instance of it).

Anyways if I'm having trouble describing what I mean... I actually got it working so this should clear it up:

Python code:

In [1]: from smartcompose import delegate

In [2]: @delegate('_n', ('__add__', '__mul__'))
   ...: class NumberWrapper:
   ...:     def __init__(self, n):
   ...:         self._n = n
   ...:

In [3]: n = NumberWrapper(10)

In [4]: n + 2
Out[4]: 12

In [5]: n * 3
Out[5]: 30

github

# ? Jun 4, 2014 16:57

Haystack: Jan 23, 2005

What's the use case you have in mind?

# ? Jun 4, 2014 20:29

SurgicalOntologist: Jun 17, 2004

Well twice recently I have used a similar design pattern--though I haven't gone back and changed it to use this trick because I'm not sure how robust my implementation is. I wouldn't be at all surprised if there was a better way to design it in both cases, but anyways...

The first, and primary use case I had in mind, is for a library I've been working on which has a tree as its primary data structure. It's for designing experiments, and an experiment is represented as a tree with named levels--the experiment contains participants which contain sessions which contain blocks which contain trials, for example. So I found myself writing code like for participant in experiment.children until I learned about magic methods, when I changed it so I could write for participant in experiment. That required writing a few methods like:

Python code:

def __iter__(self):
    return self.children.__iter__()

def __len__(self):
    return self.children.__len__()

(Yes, in actuality I used iter and len on self.children instead of the dunder methods, but that illustrates well the symmetry). That violated my DRY sensibilities--I wanted a way to say "self.children is in charge of the sequence protocol for self".

The second time it came up was writing code that interfaced between a 2D geometry library and pyglet--because pyglet has sprites but not simple shapes like rectangles. Someone somewhere convinced me not to inherit from the geometry library's Polygon class but instead use composition. Which maybe was a mistake, because I soon found myself in a similar situation. I wanted to be able to, for example, transpose my shape with in-place addition and resize it with in-place multiplication, like Polygon did. So again I found myself writing repetitive code like

Python code:

def __iadd__(self, other):
    return self.polygon.__iadd__(other)

An idea for further improvement would be to have a dictionary that maps protocol names to sets of methods. So you could use @delegate('_stuff', protocols['sequence']) instead of [fixed]@delegate('_stuff' ('__len__', '__getitem__', '__setitem__', '__delitem__')) or whatever.

SurgicalOntologist fucked around with this message at 02:42 on Jun 5, 2014

# ? Jun 5, 2014 02:36

BigRedDot: Mar 6, 2008

Megaman posted:

Can you elaborate? setuptools is required for lots of things as far as I know, no? What would replace it and why?

One of our founders puts it pretty well:

quote:

Fundamentally, distutils conflates several things: build specification, package metadata, and installation specification. To make this even worse, it stuck all these *inside an imperative executable* called setup.py. It's like taking a swarm of angry hornets and arming them with nukes. There are only a small, finite number of ways to make this significantly worse.

The problem was that once you installed any python package with distutils, it would splat out all over your Python directory, possibly across /usr/lib and everything else. The desire for a more modular Python package format arose out of the need for a plugin system in the Chandler PIM software, which was being developed with Python. The setuptools library arose from this, and invented the concept of an "egg". The problem is that setuptools managed to pick one of those small, finite ways to make distutils worse - in fact it could be argued that it took several. Not only was it abandoned for a long time in a half-finished state, but it introduced *just enough* features to be kind of useful: binary packages, automatic install of versioned dependencies, "setup.py develop", etc. But virtually everything that setuptools did, it did in a completely half-assed way. Automatic installs of versioned dependencies would crap out halfway through, and not tell you why it broke. Etc. Trying to do *simple* things required all sorts of obscure magic, including my favorite command line option of all time, --single-version-externally-managed. Any other sane implementer of a unix command line tool would have just called that option "-f". But no. I'm a little surprised Philip Eby didn't go all the way and just call it --override-default-value-of-argument-foo-to-function-bar-in-source-file-baz, which would at least have been more explicit. Hmm, maybe that's why he chose not to call it that.

Since setuptools had so many warts and was so frustrating to use, "pip" was introduced. Which basically did very little, except to wrap some of those warts with nicer options. Under the covers, it basically did --single-version-externally-managed for *all packages*, and also then provided the basis for tools like virtualenv. But pip itself was just a wrapper/hack on setuptools. People "upgraded" to it from setuptools because of the fact that 99.99999% of people don't do anything more than "pip install <package>". They really don't.

Then attempts were made to excise the tumor of setuptools from Python, via the distribute and then distutils2 projects. And now pip is sort of being blessed as the canonical way to install things, even while recognizing that it does not solve the problem for a huge number of very important extension modules.

And there you have the Python packaging saga in a nutshell.

And some images from a recent keynote:

The only marginally useful feature I found of setuptools was "python setup.py develop" but you can reproduce that by hand with five lines of code in your setup.py which is exactly what we did for Bokeh, rather than use setuptools. As for a replacement, naturally I will suggest taking a look at conda, but of course I am biased (I wrote the first version of conda two years ago, it's been taken over by others and taken much further since then).

BigRedDot fucked around with this message at 03:58 on Jun 5, 2014

# ? Jun 5, 2014 03:47

Comrade Gritty: Sep 19, 2011; This Machine Kills Fascists

You should use setuptools. In fact if you use pip your setup.py is going to be forced to use setuptools whether you import it or not.

Yes setuptools sucks, distutils sucks more. We're slowly unraveling this great big ball of poo poo and it's getting better and a lot of that is going to be shimming the new stuff in via setuptools. One of the biggest problem with distutils is that it's impossible to actually fix anything in it, in a way that anyone is able to actually take advantage of in the next 20 years. Even if we waved our hands and made ti the perfect system, the earliest it would land is 3.5, which means no one could depend on it unless 3.5 was the minimum version for it.

On the other hand, gently caress easy_install and everything it stands for.

# ? Jun 5, 2014 06:40

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Every time I want to package something it feels like a nightmare all over again.

# ? Jun 5, 2014 07:57

tef: May 30, 2004; -> some l-system crap ->

BigRedDot posted:

And some images from a recent keynote:

For some reason I skipped all of this and used vendor packaging formats whenever I had to deploy python.

Anyway, is conda planning to support or use wheels internally?

# ? Jun 5, 2014 15:37

Plorkyeran: Mar 22, 2007; To Escape The Shackles Of The Old Forums, We Must Reject The Tribal Negativity He Endorsed

Bundling everything in distro-specific packages is generally the right way to handle deployment to single-purpose linux servers regardless of what you're deploying, but doesn't really work for anything else.

# ? Jun 5, 2014 15:44

namaste friends: Sep 18, 2004; by Smythe

What do you guys think of using docker to package your python app? I know it's overkill but in the interest of getting stuff done, it seems like the cleanest way.

# ? Jun 5, 2014 16:52

BigRedDot: Mar 6, 2008

tef posted:

For some reason I skipped all of this and used vendor packaging formats whenever I had to deploy python.

Anyway, is conda planning to support or use wheels internally?

Conda will interoperate with wheels to some degree, but I am not sure to what extent. The conda and wheel package formats are quite similar but not identical. That would be a great question for the conda mailing list, though.

tef posted:

For some reason I skipped all of this and used vendor packaging formats whenever I had to deploy python.

ployrkeran posted:

Bundling everything in distro-specific packages is generally the right way to handle deployment to single-purpose linux servers regardless of what you're deploying, but doesn't really work for anything else.

This is exactly the primary motivating use case that drove conda development: reproducible, cross-platform deployments (and not just of python). The best description I've heard is "cross platform homebrew"

# ? Jun 5, 2014 20:45

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

Python code:

import tempfile
from PIL import Image
import requests

image_url = 'http://static.somethingawful.com/img/logo.png'
max_size_in_bytes = 10 * 1024 * 1024

r = requests.get(image_url, stream=True)
with tempfile.TemporaryFile() as f:
    size = 0
    for chunk in r.iter_content(chunk_size=1024):
        #filer out keep-alive chunks
        if chunk:
            f.write(chunk)
            size += len(chunk)
            if size > max_size_in_bytes:
                raise Exception('Too big')
    image = Image.open(f)
    print image.format

How come this throws IOError: cannot identify image file on the Image.open call?

# ? Jun 6, 2014 01:41

Gangsta Lean: Dec 3, 2001; Calm, relaxed...what could be more fulfilling?

Your file isn't closed. Unindent the last two lines of your program so that they are outside the scope of the "with" line (which automatically closes your file when the block's scope ends).

Edit: But you'd have to open the file again, first. Or do f.seek(0) before Image.open(f) I guess.

Gangsta Lean fucked around with this message at 02:03 on Jun 6, 2014

# ? Jun 6, 2014 02:01

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

Gangsta Lean posted:

Your file isn't closed. Unindent the last two lines of your program so that they are outside the scope of the "with" line (which automatically closes your file when the block's scope ends).

Edit: But you'd have to open the file again, first. Or do f.seek(0) before Image.open(f) I guess.

Ok that does make sense. f.seek(0) didn't seem to do it though. And how do I re-open it if it's a TemporaryFile? Do I have to use a NamedTemporaryFile and hang on to the name of it?

edit: NamedTemporaryFile doesn't seem to work either, the file is already gone before it gets to Image.open it

fletcher fucked around with this message at 02:17 on Jun 6, 2014

# ? Jun 6, 2014 02:08

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

This seemed to work...am I leaking file descriptors?

Python code:

import tempfile
from PIL import Image
import requests

image_url = 'http://static.somethingawful.com/img/logo.png'
max_size_in_bytes = 10 * 1024 * 1024

temp_file = tempfile.NamedTemporaryFile()

r = requests.get(image_url, stream=True)
with open(temp_file.name, 'w') as f:
    size = 0
    for chunk in r.iter_content(chunk_size=1024):
        #filer out keep-alive chunks
        if chunk:
            f.write(chunk)
            size += len(chunk)
            if size > max_size_in_bytes:
                raise Exception('Too big')

image = Image.open(temp_file.name)
print image.format

# ? Jun 6, 2014 02:26

Crosscontaminant: Jan 18, 2007

Doesn't look like it to me - the context manager handles everything when you exit the scope of the with clause, then PIL (p.s. use Pillow if you aren't already, it's actually maintained) will close its own handle.

Also, OverflowError exists, and is more descriptive than throwing a generic Exception.

# ? Jun 6, 2014 10:25

Dominoes: Sep 20, 2007

Unit tests: What's an ideal place to put overall setup code? Ie loading a large database that multiple tests use. SetUp() won't do, since that's executed for each test. Are globals the right answer, or is there an ideal way to insert it as a unittest.TestCase class var?

Dominoes fucked around with this message at 11:16 on Jun 6, 2014

# ? Jun 6, 2014 10:46

tef: May 30, 2004; -> some l-system crap ->

make a helper method or function, make it a context manager, and use it in a with statement?

# ? Jun 6, 2014 11:11

Surprise T Rex: Apr 9, 2008; Dinosaur Gum

Not really specifically a Python question, but what are some good resources for learning how to actually design OOP stuff? I know the why, what and technically how, but I think I basically still program in an imperative way but with classes and functions. Like, it's broken up, but I'm not sure it's reusable.

# ? Jun 6, 2014 11:38

nonathlon: Jul 9, 2004; And yet, somehow, now it's my fault ...

BigRedDot posted:

The only marginally useful feature I found of setuptools was "python setup.py develop" but you can reproduce that by hand with five lines of code in your setup.py which is exactly what we did for Bokeh, rather than use setuptools. As for a replacement, naturally I will suggest taking a look at conda, but of course I am biased (I wrote the first version of conda two years ago, it's been taken over by others and taken much further since then).

What are those 5 lines? (As I adored working with "python setup.py develop".)

Setuptools always worked nicely for me - but that's as a user who never had to delve into the innards of the package.

# ? Jun 6, 2014 11:57

good jovi: Dec 11, 2000

Dominoes posted:

Unit tests: What's an ideal place to put overall setup code? Ie loading a large database that multiple tests use. SetUp() won't do, since that's executed for each test. Are globals the right answer, or is there an ideal way to insert it as a unittest.TestCase class var?

If you need to set up a database, then you're not writing unit tests. Stub that stuff out.

That said, unittest provides module-level and class-level setup/teardown hooks. Nose will respect those, and I think it might even have a package-level setUp. Nose2 has a concept of layers that I haven't used, but looks weird.

# ? Jun 6, 2014 13:22

Lysidas: Jul 26, 2002; John Diefenbaker is a madman who thinks he's John Diefenbaker.; Pillbug

fletcher posted:

This seemed to work...am I leaking file descriptors?

Python code:

import tempfile
from PIL import Image
import requests

image_url = 'http://static.somethingawful.com/img/logo.png'
max_size_in_bytes = 10 * 1024 * 1024

temp_file = tempfile.NamedTemporaryFile()

r = requests.get(image_url, stream=True)
with open(temp_file.name, 'w') as f:
    size = 0
    for chunk in r.iter_content(chunk_size=1024):
        #filer out keep-alive chunks
        if chunk:
            f.write(chunk)
            size += len(chunk)
            if size > max_size_in_bytes:
                raise Exception('Too big')

image = Image.open(temp_file.name)
print image.format

Open your file in binary mode for writing ('wb') too.

# ? Jun 6, 2014 14:51

BigRedDot: Mar 6, 2008

outlier posted:

What are those 5 lines? (As I adored working with "python setup.py develop".)

Something like:

code:

path_file = join(getsitepackages()[0], "bokeh.pth")

if 'develop' in sys.argv:
    with open(path_file, "w+") as f:
        f.write(path)
    sys.exit()

elif 'install' in sys.argv:
    if exists(path_file):
        os.remove(path_file)

# ? Jun 6, 2014 16:32

nonathlon: Jul 9, 2004; And yet, somehow, now it's my fault ...

BigRedDot posted:

Something like:

code:

path_file = join(getsitepackages()[0], "bokeh.pth")

if 'develop' in sys.argv:
    with open(path_file, "w+") as f:
        f.write(path)
    sys.exit()

elif 'install' in sys.argv:
    if exists(path_file):
        os.remove(path_file)

Makes complete sense.

While I'm here, a regex problem. I have a complex dataset extracted from a database as a csv file. The headers of the columns are written as "Foo(Bar)(Baz)" where Foo is a field in the Bar subsection of the Baz section. I'm trying to rewrite the headers as something more sensible, e.g. Baz_Foo_Bar, Baz.Bar.Foo, etc.

Problems:

- there's a variable number of levels. E.g. some headers are just "Foo", others are "Foo(Bar)(Baz)(Quux)".
- even more problematic, some genius named a bunch of the fields things like "Bar(x)", or "Baz(Plugh)" . So you end up with headers like "Foo(Bar(x))(Baz(Phlugh))"
- there is zero change of getting a different or better formatted output. While it's our data, the db is proprietary and in someone else's hands.

Ideas on how to split these suckers?

# ? Jun 6, 2014 17:50

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

Crosscontaminant posted:

Doesn't look like it to me - the context manager handles everything when you exit the scope of the with clause, then PIL (p.s. use Pillow if you aren't already, it's actually maintained) will close its own handle.

Also, OverflowError exists, and is more descriptive than throwing a generic Exception.

Lysidas posted:

Open your file in binary mode for writing ('wb') too.

Very helpful! Thanks guys!

# ? Jun 6, 2014 19:24

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

vikingstrike posted:

Here are the docs for a Series sort: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.sort.html?highlight=sort#pandas.Series.sort
and the DataFrame sort: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sort.html?highlight=sort#pandas.DataFrame.sort

If your DataFrame is df, then just use df.sort(['var1', 'var2']). For Series, it should just be s.sort().

I know this is from forever ago, but these (and .order) are even more hilariously broken.

unrelated, but has GitHub never had the ability to just search bugs or have I just never reported a bug on a project with hundreds of bugs?

# ? Jun 6, 2014 22:13

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

Munkeymon posted:

I know this is from forever ago, but these (and .order) are even more hilariously broken.

unrelated, but has GitHub never had the ability to just search bugs or have I just never reported a bug on a project with hundreds of bugs?

Use the search at the top of the page (it should be set to 'this repository') and then there will be multiple categories of results you can flip through on the left side (i.e. Code, Issues, etc)

# ? Jun 6, 2014 22:36

namaste friends: Sep 18, 2004; by Smythe

Can someone explain to me what's going on when you call a method like this:

code:

import boto.ec2
ec2 = boto.ec2.connect_to_region('us-west-2)
group = conn.get_all_groups(names=['my_group'])[0]
instance_ids = [i.instance_id for i in group.instances]
instances = ec2.get_only_instances(instance_ids)

This is from here: http://docs.pythonboto.org/en/latest/autoscale_tut.html

What does some.method()[0] do exactly?

e: nm I figured it out. conn.get_all_groups returns a list. conn.get_all_groups()[0] returns the first element of that list.

namaste friends fucked around with this message at 07:08 on Jun 7, 2014

# ? Jun 7, 2014 07:02

ohgodwhat: Aug 6, 2005

Munkeymon posted:

I know this is from forever ago, but these (and .order) are even more hilariously broken.

I've never had an issue with pd's sorts, which isn't to say it can't happen, I'm just curious what you're doing that you're seeing a problem.

# ? Jun 7, 2014 19:40

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 20:19

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I just want to make sure that I'm not missing something...

There's no way in PySide/PyQT to set global hotkey shortcuts? By global, I mean hotkeys that trigger even when your GUI doesn't have focus.

I can't find a way to do it from PySide, and the other two things I find don't work for my needs. PyHK has an incompatible license, and PyGS hasn't been updated in 4 years and doesn't work or compile.

I'm at a tipping point where I'm going to either wire up some monstrosity between AutoIT and my GUI or scrap this project.

# ? Jun 9, 2014 18:09

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »