Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

onionradish: Jul 6, 2006; That's spicy.

is right -- for f's sake -- I picked lxml thinking it would be a stupidly simple test package! I'm torn between being happy it's not my fault and angry about the hoops I'm going to have to jump through....

I'll give the VS 2008 and "manually-copy-the-binary" methods a try.

# ? Dec 14, 2012 22:48

Adbot: ADBOT LOVES YOU

# ? May 31, 2024 01:09

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

onionradish posted:

Following up on virtualenv, I'm having failures when trying to install packages, and I'm not sure what I'm doing wrong.

I'm on Windows, and using CMD as the shell. I'm able to create a project folder: "virtualenv test". After I do that, I'm navigating to the test directory (D:\test\project) and entering "scripts\activate". Then I'm trying "pip install lxml" as an example.

A bunch of stuff scrolls by and results in "failed with error code 1". I'm not sure what this means:
code:
Command D:\test\project\Scripts\python.exe -c "import setuptools;__file__='
c:\\windows\\temp\\pip-j7v8xg-build\\setup.py';exec(compile(open(__file__).read(
).replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\windows\temp\pi
p-q2edge-record\install-record.txt --single-version-externally-managed --install
-headers D:\test\project\include\site\python2.7 failed with error code 1 in
 c:\windows\temp\pip-j7v8xg-build
For what it's worth, I don't see "D:\test\project\include\site\*" as a folder; the "include" path only has a bunch of *.h files. What am I doing wrong?

Welcome to one of the large reasons I switched to linux on the desktop. Just got tired of trying to find windows binaries or getting VS2008 to compile stuff. VS2008 will work for some stuff and not for others, for reasons that I got tired of sussing out.

# ? Dec 14, 2012 23:53

Captain Capacitor: Jan 21, 2008; The code you say?

Thermopyle posted:

Welcome to one of the large reasons I switched to linux on the desktop. Just got tired of trying to find windows binaries or getting VS2008 to compile stuff. VS2008 will work for some stuff and not for others, for reasons that I got tired of sussing out.

To be fair, the package managers are the saving grace for Linux when it comes to libraries. There are some libraries with really specific version requirements that need apt-pinning or other such juggling.

Not going to deny it's a lot easier overall, though.

# ? Dec 15, 2012 00:28

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

I spent a few hours trying to compile PyCrypto on Windows before giving up and finding binaries. It was incredibly frustrating.

# ? Dec 15, 2012 04:11

Janitor Prime: Jan 22, 2004; PC LOAD LETTER

What da fuck does that mean; Fun Shoe

BeefofAges posted:

I spent a few hours trying to compile PyCrypto on Windows before giving up and finding binaries. It was incredibly frustrating.

Why is everyones first instinct to try and compile it instead of just installing the binaries?

# ? Dec 15, 2012 17:01

Captain Capacitor: Jan 21, 2008; The code you say?

Because "pip install pycrypto" is easier than Googling.

# ? Dec 15, 2012 17:31

onionradish: Jul 6, 2006; That's spicy.

Hard NOP Life posted:

Why is everyones first instinct to try and compile it instead of just installing the binaries?

Wait, *can* you install binaries to virtualenv setups? I'd far rather do that, and it's how I normally install stuff to the root Python folder. Dealing with compilation sucks.

edit: ~~I'll be damnned; looks like you *can* install most binaries to virtualenv. Time for some testing...~~ Yeah, no, that doesn't actually work.

onionradish fucked around with this message at 18:51 on Dec 15, 2012

# ? Dec 15, 2012 18:29

Janitor Prime: Jan 22, 2004; PC LOAD LETTER

What da fuck does that mean; Fun Shoe

Captain Capacitor posted:

Because "pip install pycrypto" is easier than Googling.

Except on windows which is what we're discussing.

# ? Dec 15, 2012 18:31

onionradish: Jul 6, 2006; That's spicy.

An update to my earlier post: the linked StackOverflow post was about using "easy_install" in a Windows virtualenv, but testing with binaries for PIL, it only appears to work, meaning it actually doesn't. The package only gets partially installed. The same SO post includes a suggestion to change the registry around, which seems really hacky.

onionradish fucked around with this message at 18:53 on Dec 15, 2012

# ? Dec 15, 2012 18:50

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Captain Capacitor posted:

Because "pip install pycrypto" is easier than Googling.

Basically this. It works for almost everything else, so when you get to PyCrypto, you try to install it with pip and get errors. When you google the errors, you get various convoluted guides on how to make Windows compilation work instead of getting links to binaries. When you go to the PyCrypto homepage, there isn't even any mention of binaries.

Basically it's a documentation issue.

# ? Dec 15, 2012 19:44

Vulture Culture: Jul 14, 2003; I was never enjoying it. I only eat it for the nutrients.

duck monster posted:

Use this thing: http://kivy.org/

It makes incredibly cool UIs.

Thanks for this, it looks perfect for the PyKaraoke megafork I'm working on.

# ? Dec 15, 2012 21:13

Captain Capacitor: Jan 21, 2008; The code you say?

BeefofAges posted:

Basically this. It works for almost everything else, so when you get to PyCrypto, you try to install it with pip and get errors. When you google the errors, you get various convoluted guides on how to make Windows compilation work instead of getting links to binaries. When you go to the PyCrypto homepage, there isn't even any mention of binaries.

Basically it's a documentation issue.

I usually have it crop up as a nested requirement for some library (I'm looking at you, Twisted :argh:

)

# ? Dec 15, 2012 22:16

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Yeah, I needed it for paramiko.

# ? Dec 15, 2012 22:22

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Python package distribution on Windows is just awful. I had to unzip some egg files in some magical location because the binary installer that had an InstallShield blue background and "PYTHON POWERED" in some dots text and told me that I had 205GB disk space free on C: couldn't find that I had Python 2.7.

A part of it now is that MS's compiler suite is really awful and breaks between versions and isn't freely available without installing some monster Visual C++ IDE that I don't want and will never use, because the version bundled in the Windows 7 SDK is just flat broken.

I don't know what the solution is, but they need to make "pip install pycrypto" just work.

# ? Dec 16, 2012 00:13

Movac: Oct 31, 2012

I haven't tested this, but it looks like distutils can use gcc via MinGW. It seems like MinGW is the preferred way to get a C compiler on Windows for open-source work, so that's probably your best option.

# ? Dec 16, 2012 00:24

JetsGuy: Sep 17, 2003; science + hockey
=
LASER SKATES

I'm sure there's a great reason why this works the way it does, so I guess that's why I'm asking here.

When making a simple dictionary, like this:

code:

master = {}
for i in range(1,4,1):
    d_set = {"x": [], "x_err" : [], "y" : [], "y_err" : []}
    key = "set_"+str(i)
    master[key] = d_set

I can then add data to this easily, and in the way intended, ie:

code:

In [91]: master["set_1"]["x"] = 3

In [92]: master
Out[92]: 
{'set_1': {'x': 3, 'x_err': [], 'y': [], 'y_err': []},
 'set_2': {'x': [], 'x_err': [], 'y': [], 'y_err': []},
 'set_3': {'x': [], 'x_err': [], 'y': [], 'y_err': []}}

However, if I move that line defining d_set OUTSIDE the loop to create the "master" dictionary, all of a sudden I can't add/edit the dictionary the way that makes sense to me.

So like if I tried:

code:

In [91]: master["set_1"]["x"] = 3

In [92]: master
Out[92]: 
{'set_1': {'x': 3, 'x_err': [], 'y': [], 'y_err': []},
 'set_2': {'x': 3, 'x_err': [], 'y': [], 'y_err': []},
 'set_3': {'x': 3, 'x_err': [], 'y': [], 'y_err': []}}

This seems dumb as hell to me. I am assuming it's happening because I never am redefining "d_set", and so the dictionary somehow assumes to set all "x" for 3 in that case.

I just don't understand why that would be...

# ? Dec 16, 2012 01:22

Jewel: May 2, 2009

JetsGuy posted:

However, if I move that line defining d_set OUTSIDE the loop to create the "master" dictionary, all of a sudden I can't add/edit the dictionary the way that makes sense to me.

This seems dumb as hell to me. I am assuming it's happening because I never am redefining "d_set", and so the dictionary somehow assumes to set all "x" for 3 in that case.

I just don't understand why that would be...

Because python likes to use references rather than copies for everything. You're saying master["set_1"], master["set_2"], and master["set_3"] are all set to the same instance of d_set, which means when you change one of them, all of them change.

Python has a copy module, which you can use here.

Python code:

import copy
master = {}
d_set = {"x": [], "x_err" : [], "y" : [], "y_err" : []}
for i in range(1,4,1):
    key = "set_"+str(i)
    master[key] = copy.copy(d_set)

code:

In [91]: master["set_1"]["x"] = 3

In [92]: master
Out[92]: 
{'set_1': {'x': 3, 'x_err': [], 'y': [], 'y_err': []},
 'set_2': {'x': [], 'x_err': [], 'y': [], 'y_err': []},
 'set_3': {'x': [], 'x_err': [], 'y': [], 'y_err': []}}

Edit: An example of what I meant:

Python code:

def change_stuff(x):
	x[5] = "changed"

x = range(1, 10)
change_stuff(x)
print x

#output: [1, 2, 3, 4, 5, 'changed', 7, 8, 9]

You're passing a reference to the object here, unlike most(?) other languages where you're passing a copy unless specified.

Jewel fucked around with this message at 02:23 on Dec 16, 2012

# ? Dec 16, 2012 02:19

Cat Plus Plus: Apr 8, 2011

Jewel posted:

Python has a copy module, which you can use here.

You want to use deepcopy if you do it this way, otherwise:

code:

In [4]: for i in range(1, 4): master['set_' + str(i)] = copy.copy(d_set)

In [5]: master['set_1']['x'].append(42)

In [6]: master
Out[6]:
{'set_1': {'x': [42], 'x_err': [], 'y': [], 'y_err': []},
 'set_2': {'x': [42], 'x_err': [], 'y': [], 'y_err': []},
 'set_3': {'x': [42], 'x_err': [], 'y': [], 'y_err': []}}

But it's better to just keep the literal in the loop.

# ? Dec 16, 2012 02:35

Scaevolus: Apr 16, 2007

Jewel posted:

Python has a copy module, which you can use here.

As mentioned, you want to use deepcopy for that.

If you want a shallow copy of a builtin type, create a new object with it as the parameter.

code:

dictcopy = dict(olddict)
setcopy = set(oldset)
listcopy = list(oldlist)

# ? Dec 16, 2012 02:49

Jewel: May 2, 2009

PiotrLegnica posted:

You want to use deepcopy if you do it this way, otherwise:

code:

In [4]: for i in range(1, 4): master['set_' + str(i)] = copy.copy(d_set)

In [5]: master['set_1']['x'].append(42)

In [6]: master
Out[6]:
{'set_1': {'x': [42], 'x_err': [], 'y': [], 'y_err': []},
 'set_2': {'x': [42], 'x_err': [], 'y': [], 'y_err': []},
 'set_3': {'x': [42], 'x_err': [], 'y': [], 'y_err': []}}

But it's better to just keep the literal in the loop.

I can never remember whether to use copy or deep copy. Bluh.

# ? Dec 16, 2012 03:43

The Gripper: Sep 14, 2004; i am winner

Movac posted:

I haven't tested this, but it looks like distutils can use gcc via MinGW. It seems like MinGW is the preferred way to get a C compiler on Windows for open-source work, so that's probably your best option.

I'm almost certain you have to compile everything python-related with VS 2008, including extensions, unless you compile your local python with a different compiler (and then you need to compile everything else with that).

e; oh I guess it says that.

# ? Dec 16, 2012 05:06

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

What's the advantage of using deepcopy here, which has to go through a crazy pickle round-trip, rather than just keeping the line inside the for loop?

# ? Dec 16, 2012 05:25

Jewel: May 2, 2009

Suspicious Dish posted:

What's the advantage of using deepcopy here, which has to go through a crazy pickle round-trip, rather than just keeping the line inside the for loop?

I guess it's more so semantics on why what happens, happens, and how to fix it, for a future reference.

# ? Dec 16, 2012 05:28

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

A rule of thumb for me is that importing the copy module means you're doing something wrong.

# ? Dec 16, 2012 05:42

Jewel: May 2, 2009

Suspicious Dish posted:

A rule of thumb for me is that importing the copy module means you're doing something wrong.

Copy could be cool for stuff like, say, passing a default Particle type into a ParticleEmitter. Of course you could then argue that you should pass in settings to create a particle rather than passing in a pre-made particle, but it could be useful sometimes!

Edit: Yeah nevermind that's a goddamn terrible example. I'm sure there's some reasons to use it though.

# ? Dec 16, 2012 05:51

tef: May 30, 2004; -> some l-system crap ->

Suspicious Dish posted:

A rule of thumb for me is that importing the copy module means you're doing something wrong.

Similarly, I don't use copy, but I use slices a lot.

I often find myself doing foo = {}; foo.update(bar); ....

# ? Dec 16, 2012 06:24

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

tef posted:

Similarly, I don't use copy, but I use slices a lot.

I often find myself doing foo = {}; foo.update(bar); ....

Is foo = dict(bar) too non-obvious?

# ? Dec 16, 2012 06:57

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Jewel posted:

Copy could be cool for stuff like, say, passing a default Particle type into a ParticleEmitter. Of course you could then argue that you should pass in settings to create a particle rather than passing in a pre-made particle, but it could be useful sometimes!

Edit: Yeah nevermind that's a goddamn terrible example. I'm sure there's some reasons to use it though.

Pass in a factory function that creates a particle. This is a common pattern.

# ? Dec 16, 2012 06:59

tef: May 30, 2004; -> some l-system crap ->

Suspicious Dish posted:

Is foo = dict(bar) too non-obvious?

I do that a lot too. I use update when i'm adding things in an if-chain somewhere down the line.

Edit: According to grep, I use foo = dict(bar) most of the time.

# ? Dec 16, 2012 07:12

Scaevolus: Apr 16, 2007

Suspicious Dish posted:

Pass in a factory function that creates a particle. This is a common pattern.

For example,

Python code:

from collections import defaultdict

wordcount = defaultdict(lambda: 0)
wordcount = defaultdict(int)  # identical

wordcount["the"] += 1  # missing key's value is initialized to 0

The copy module has to do many checks, so it should be avoided whenever possible.

# ? Dec 16, 2012 11:30

Cat Plus Plus: Apr 8, 2011

copy module is useful for two things:
- generic code that wants to do copies while preserving types
- deep copying, when it's actually needed (sometimes it is, but that's not very common use case)

Using type constructor directly is better if the type is known upfront, or you want to copy and convert.

# ? Dec 16, 2012 17:38

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

I don't mind using copy or deepcopy when it's convenient, since the code I write is almost never a performance bottleneck. I mostly work in test automation, so the scripts I write usually spend most of their time waiting for other software to do stuff.

# ? Dec 16, 2012 21:05

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

It's not a performance issue. The code that copies custom classes that you write are copied in weird ways (it doesn't call your constructor) that can cause your code to suddenly break.

# ? Dec 16, 2012 21:08

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Suspicious Dish posted:

It's not a performance issue. The code that copies custom classes that you write are copied in weird ways (it doesn't call your constructor) that can cause your code to suddenly break.

Oh, that makes sense. I've never actually used it on custom classes, just on dicts full of standard types.

# ? Dec 16, 2012 21:11

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

I recently interviewed for a python engineer position and got some questions that were a bit different from what I'm used to at programming interviews. Usually I get asked to write a sorting function, or a linked list implementation, or to find the bugs in a code snippet. I've even been asked to figure out what complexity class a given algorithm is in. However, at this interview, I got asked really specific things, like what the syntax is for list comprehensions, or how to write a generator, or what the difference is between '==' and 'is'. The only actual coding I was asked to do was to write a script that parses a given web page and makes its contents available through a REST API that serves JSON. They gave me a laptop and left me alone for an hour and said I could use whatever resources I wanted. I've never actually done any web development or HTML parsing or worked with REST or JSON, so I didn't actually get farther than some basic parsing using examples copied and pasted from stackoverflow. I'm still trying to decide whether the stuff they asked me is useful for separating good programmers from bad ones or not.

Funny thing is, they offered me the job, but only offered about 2/3rds of what I earn now, even after I told them what I'm making right now. I ended up turning them down for reasons unrelated to the salary they offered.

# ? Dec 16, 2012 22:10

Lurchington: Jan 2, 2003; Forums Dragoon

the == and 'is' difference is certainly important (all dynamic languages have their equivalent, and not knowing that there's a different is a big gotcha) and since a lot of python out there is for web server processes, I'm not surprised about the practical programming question.

The other stuff seems aimed at identifying "python programmers" vs "dynamic language programmers" and although that's fine, hard to say if it's serving up better candidates.

edit: I use a LOT of copy.deepcopy() in unittest code that wants to test nested dictionaries (e.g. JSON) with slight differences.

code:

butts = dict(<lots of keys with nested dicts>)
dongs = copy.deepcopy(butts)
dongs.pop('some key')

self.assertEqual(expected, my_function(butts))
self.assertEqual(other_expected, my_function(dongs))

so I can add new stuff to butts and dongs has it, and I can add or remove things from dongs without messing up butts

Lurchington fucked around with this message at 00:56 on Dec 17, 2012

# ? Dec 17, 2012 00:50

Opinion Haver: Apr 9, 2007

Now I'm wondering if there are dynamic languages with immutable values/purity a la Haskell so that you don't need to distinguish between == and is.

# ? Dec 17, 2012 01:12

ShoulderDaemon: Oct 9, 2003; support goon fund; Taco Defender

yaoi prophet posted:

Now I'm wondering if there are dynamic languages with immutable values/purity a la Haskell so that you don't need to distinguish between == and is.

Lambda calculus.

# ? Dec 17, 2012 02:21

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

How do I use SQLAlchemy's declared_attr with Flask-SQLAlchemy?

Specifically, I'm trying to implement this pattern (taken from this):

Python code:

import re
from sqlalchemy.ext.declarative import declared_attr
class Base(object):

    @declared_attr
	def __tablename__(cls):
        # convert from CamelCase to words_with_underscores
        name = cls.__name__
        return (
            name[0].lower() +
            re.sub(r'([A-Z])', 
            lambda m:"_" + m.group(0).lower(), name[1:])
        )

	# provide an "id" column to all tables
	id = Column(Integer, primary_key=True)

Base = declarative_base(cls=Base)

This question may or may not make sense...I'm a little shaky in my understanding of what the point of Flask-SQLAlchemy actually is. Why shouldn't I just use SQLAlchemy instead of the Flask extension?

# ? Dec 17, 2012 02:37

Adbot: ADBOT LOVES YOU

# ? May 31, 2024 01:09

Haystack: Jan 23, 2005

No, your question makes perfect sense. SQLalchemy's declarative bases are a really nice feature. Unfortunately, Flask-SQLalchemy appears to cover that feature up, to some extent.

As far as what Flask-SQLalchemy offers, the relevant bit of documentation is refreshingly clear about that. Essentially it:

Auto-magically configures some important SQLalchemy components based on some relevant Flask app configuration settings.
Offers a nice centralized place access some of SQLalchemy's core types.
Cleans up the database session at the end of a request, which is VERY nice.
Offers a preconfigured SQLa Base class called Model, which offers some nice convenience methods for querying without futzing around with the database session.

The Model class is where you'll run into trouble: the way it is generated makes it difficult to extend.

Nonetheless, here's my hacky untested attempt at doing it anyway!

Python code:

import re
from flask import Flask
from flask.ext.sqlalchemy import * #Because I'm lazy
from sqlalchemy.ext.declarative import declarative_base, declared_attr

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)


class MyModel(Model):

    @declared_attr
    def __tablename__(cls):
        # convert from CamelCase to words_with_underscores
        name = cls.__name__
        return (
            name[0].lower() +
            re.sub(r'([A-Z])', 
            lambda m:"_" + m.group(0).lower(), name[1:])
        )

    # provide an "id" column to all tables
    id = Column(Integer, primary_key=True)

#Torn from the bloody, beating heart of the Flask-SQLalchemy source
base = declarative_base(cls=MyModel, name='Model',
                        mapper=signalling_mapper,
                        metaclass=_BoundDeclarativeMeta)
base.query = BaseQuery(db)
db.MyModel = base #Use like you would use db.model

# ? Dec 17, 2012 05:11

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »