Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

brosmike: Jun 26, 2009

Jam2 posted:

In what sort of environment do you guys code Python? I've been juggling notepad++ and the Python shell, pasting code across and evaluating it to see when it breaks. Is there a better way? I watched a Google lecture on Python. The speaker utilized Terminal and executed Python in that way. How would I do this in Windows? Should I work in OS X?

My editor of choice is currently PyCharm, but for smaller projects I usually switch between a python command line and notepad++. Note that you don't have to copy-paste code, generally - just open the python command line from wherever your script is and you can do "import myscript" from within the python shell. You can then do "reload(myscript)" if you make changes and don't want to restart the whole shell.

If you want to do this, make sure that the "normal functionality" of your script is wrapped in a block that looks like this:

code:

if __name__ == '__main__':
    #your main code

If you do this, your script will still do what you want when you run it directly, but it won't run the main code when you import it into the shell.

Jam2 posted:

How do I import a text file and go about parsing the contents? Do I need to import modules to do this? I want to be able to locate certain patterns of data, create lists, strings, and assign values to variables.

You can import modules to make parsing easier if the files are formatted in particular ways (for example, there's a separate module for making it easy to read CSV files, and there are several libraries for XML parsing), but if you just want to access the text of a file line by line, the easiest way is:

code:

with open(my_filename) as my_file:
    for line in my_file:
        #line is a string, do what you want with it

This assumes that you're using a python version new enough for the with statement, I forget which version added it.

Jam2 posted:

I've been using Python 3. Is this a bad idea? Should I stick with a version that's been in production for a while? The Facebook puzzle robot accepts Python 2.5.2. A visit to the Python 2.5.2 page informs that this version has been replaced by 2.5.5. What is a newbie to do?

Python 2.5.5 is at this point pretty old, and you should avoid it in favor of 3.1 or at least 2.7 where possible. Obviously there are some situations - apparently, like the Facebook puzzle robot - where you don't have control over this. The general consensus is to use 3.1 whenever you can, and 2.7 when you need compatibility with a non-python-3-compatible library (that is, most of them).

# ? Dec 20, 2010 02:01

Adbot: ADBOT LOVES YOU

# ? Jun 3, 2024 04:38

Lurchington: Jan 2, 2003; Forums Dragoon

the reason Facebook is on Python 2.5.2 is possibly because it's backended by Jython or Google App Engine

VVV if the only reason people keep saying is 'too hard to update everything' or (even better) 'they're too lazy to update', then that's an incomplete picture.

Even gigantic infrastructures requiring 2.5.x wouldn't necessarily by stuck on 2.5.2 forever. And the exact kind of organizational behemoth that mandates software package versions would also mandate installing security updates as soon as possible. Jython and App Engine just happen to be on 2.5.2

Lurchington fucked around with this message at 15:48 on Dec 20, 2010

# ? Dec 20, 2010 02:11

king_kilr: May 25, 2007

Lurchington posted:

the reason Facebook is on Python 2.5.2 is possibly because it's backended by Jython or Google App Engine

No. It's because upgrading a large amount of infrasture is a lot of work.

# ? Dec 20, 2010 04:24

Pianist: Jun 9, 2001; NOT A RAPIST.

Jam2 posted:

In what sort of environment do you guys code Python? I've been juggling notepad++ and the Python shell, pasting code across and evaluating it to see when it breaks. Is there a better way? I watched a Google lecture on Python. The speaker utilized Terminal and executed Python in that way. How would I do this in Windows? Should I work in OS X?

I use python in both Linux and Windows at work and use emacs with python-mode in both environments. The learning curve is a little steep if you're used to standard windows/mac text editors but not daunting. It's awesome to actually having the python interpreter running the same window that you're editing in. You can easily highlight a chunk of code and execute it with a keystroke combo, and there are lots of other useful keystroke commands for controlling indentation and similar things.

# ? Dec 20, 2010 05:26

Ferg: May 6, 2007; Lipstick Apathy

Cross-posting this with the Django thread:

I'm trying to dynamically load a class from a specific module (called 'commands') and the code runs totally cool on my local setup running from a local Django server. This bombs out though when I deploy to Google App Engine. I've tried adding the commands module's parent module to the __import__ as well with no avail (on either setup in that case). Here's the code:

code:

mod = __import__('commands.%s' % command, globals(), locals(), [command])
return getattr(mod, command)

App Engine just throws an ImportError whenever it hits this.

Edit: and the clarify, it doesn't bomb out on the commands module. If I have a command like 'commands.cat' it can't find 'cat'.

Ferg fucked around with this message at 17:50 on Dec 20, 2010

# ? Dec 20, 2010 17:46

nbv4: Aug 21, 2002; by Duchess Gummybuns

Ferg posted:

Cross-posting this with the Django thread:

I'm trying to dynamically load a class from a specific module (called 'commands') and the code runs totally cool on my local setup running from a local Django server. This bombs out though when I deploy to Google App Engine. I've tried adding the commands module's parent module to the __import__ as well with no avail (on either setup in that case). Here's the code:
code:
mod = __import__('commands.%s' % command, globals(), locals(), [command])
return getattr(mod, command)
App Engine just throws an ImportError whenever it hits this.

Edit: and the clarify, it doesn't bomb out on the commands module. If I have a command like 'commands.cat' it can't find 'cat'.

GAE runs a restricted version of 2.5.whatever. For instance, the apply() built-in does not wok in GAR, yet that builtin is included in the ame version of python that GAE claims to run. It's possible that the __import__ trick you're doing is not supported in GAE's cusom implementation.

# ? Dec 21, 2010 02:41

king_kilr: May 25, 2007

nbv4 posted:

GAE runs a restricted version of 2.5.whatever. For instance, the apply() built-in does not wok in GAR, yet that builtin is included in the ame version of python that GAE claims to run. It's possible that the __import__ trick you're doing is not supported in GAE's cusom implementation.

WTF apply() doesn't work? apply is literally lambda f, *args, **kwargs: f(*args, **kwargs). In any even __import__ works, when used correctly, since Django works.

# ? Dec 21, 2010 04:40

Ferg: May 6, 2007; Lipstick Apathy

king_kilr posted:

WTF apply() doesn't work? apply is literally lambda f, *args, **kwargs: f(*args, **kwargs). In any even __import__ works, when used correctly, since Django works.

Yeah given that this is such a huge feature of Django, it has to work in some fashion. Figured it might be a 2.6/2.5 discrepancy though.

# ? Dec 21, 2010 06:41

tripwire: Nov 19, 2004; _{ghost flow}

Distutils is really making my head hurt.

Has anyone ever wrote and built c extension modules for a python project on windows? I tried installing the free version of visual studio last night and was getting nowhere trying to build this package: http://code.google.com/p/neat-python/

There are 3 modules in there which have C++ implementations, and the setup.py looks like this:

code:

from distutils.core import setup, Extension
setup(
      name='neat-python',
      version='0.1',
      description='A NEAT (NeuroEvolution of Augmenting Topologies) implementation',
      packages=['neat', 'neat/iznn', 'neat/nn', 'neat/ctrnn', 'neat/ifnn'],
      ext_modules=[
               Extension('neat/iznn/iznn_cpp', ['neat/iznn/iznn.cpp']),
               Extension('neat/nn/ann', ['neat/nn/nn_cpp/ANN.cpp', 'neat/nn/nn_cpp/PyANN.cpp']),
               Extension('neat/ifnn/ifnn_cpp', ['neat/ifnn/ifnn.cpp']),],
)

First I had to hack distutils a little because the developers apparently never foresaw that visual studio would make it past version 9, so I just hardcoded the path to the compiler right into distutils.

Then, I get this:
http://pastebin.com/6KhTWRpJ

I'm getting link errors because (and someone correct me if I'm wrong), it seems like distutils is trying to guess the names of some initialization functions by just taking the extension name and prefixing it with "init"; that may be fine for top level modules, but since these extensions sit inside a package hierarchy its passing visual studio this argument "/EXPORT initneat/ifnn/ifnn_cpp" when (I'm assuming) it should be "/EXPORT neat/ifnn/initifnn_cpp".
So, unsure of whether theres a mistake in setup.py or in distutils, I tried hacking distutils again to just export the exact name of that "init" function in the c++ source for each extension module rather than just slapping init in front of the whole name.
After doing that, I would still get a link error thrown by visual studio, but after trying 3 times in a row it presumably compiled the libraries it was complaining it didn't have and then successfully linked them.

Only when I try importing these compiled modules now, I end up with a very helpful traceback:

code:

  File "C:\Python26\Lib\site-packages\neat\iznn\__init__.py", line 11, in <module>
    from iznn_cpp import *
ImportError: DLL load failed: %1 is not a valid Win32 application.

# ? Dec 21, 2010 22:18

Yakattak: Dec 17, 2009; I am Grumpypuss
>:3

Using syslog is it possible to log to a file other than /var/log/messages or /var/log/syslog? I want my program to have it's own file :colbert:

# ? Dec 21, 2010 23:07

MaberMK: Feb 1, 2008; BFFs

Yakattak posted:

Using syslog is it possible to log to a file other than /var/log/messages or /var/log/syslog? I want my program to have it's own file

http://docs.python.org/library/logging.html

# ? Dec 22, 2010 00:29

Yakattak: Dec 17, 2009; I am Grumpypuss
>:3

MaberMK posted:

http://docs.python.org/library/logging.html

Ah poo poo that looks way better, thanks.

# ? Dec 22, 2010 00:46

Lurchington: Jan 2, 2003; Forums Dragoon

using a syslog logging handler is usually the way you need to go for any multiprocessing tasks you're doing. Logging to a file from multiple processes means errors opening the file for reading/writing

# ? Dec 22, 2010 16:02

tripwire: Nov 19, 2004; _{ghost flow}

Lurchington posted:

using a syslog logging handler is usually the way you need to go for any multiprocessing tasks you're doing. Logging to a file from multiple processes means errors opening the file for reading/writing

Not to mention a shitload of latency introduced from all that work the logging module has to do. Logging to file aside, the logging module has lots of other problems: If you have lots of uses of the function "trace" or "info", you pay for each time a string is built and passed into those functions and for the function call itself, regardless of whether the logging level would have blackholed the message anyway.
Having to call the trace() function a lot in a tight loop might not be a big deal in java where function calls are comparatively cheap, but in python that can absolutely slay your performance. So much so, that you get people pining for c preprocessor macros so they can alleviate the pain: http://dound.com/2010/02/python-logging-performance/http://dound.com/2010/02/python-logging-performance/

http://cogidev.blogspot.com/2010/02/python-tracing-performance.html

tripwire fucked around with this message at 22:02 on Dec 22, 2010

# ? Dec 22, 2010 21:58

Lurchington: Jan 2, 2003; Forums Dragoon

I actually meant logging.handlers.SysLogHandler versus a FileHandler, but you bring up some excellent points on why it might be good to use the syslog module

# ? Dec 22, 2010 22:41

Yakattak: Dec 17, 2009; I am Grumpypuss
>:3

Is there a way to configure logging (using a config file) to log errors+ to stdout and anything below that to a file?

# ? Dec 22, 2010 22:53

Carthag Tuek: Oct 15, 2005; Tider skal komme,
tider skal henrulle,
sl�gt skal f�lge sl�gters gang

Something like this (untested):

code:

import logging
import sys

logger = logging.getLogger('FancyName')

fh = logging.FileHandler('whatever.log')
fh.setFormatter(logging.Formatter("%(asctime)s - %(levelname)s - %(message)s"))
fh.setLevel(logging.DEBUG) # level that goes to file

ch = logging.StreamHandler(sys.stdout)
ch.setFormatter(logging.Formatter("%(asctime)s - %(levelname)s - %(message)s"))
fh.setLevel(logging.INFO) # level that goes to stdout

logger.addHandler(fh)
logger.addHandler(ch)

logger.debug('FancyName debug...')
logger.info('FancyName info...')

# ? Dec 23, 2010 00:53

nbv4: Aug 21, 2002; by Duchess Gummybuns

king_kilr posted:

WTF apply() doesn't work? apply is literally lambda f, *args, **kwargs: f(*args, **kwargs). In any even __import__ works, when used correctly, since Django works.

sorry i meant buffer()

# ? Dec 23, 2010 02:17

Swanson Broth: Apr 8, 2004

code:

def lprod(l):
    """ Calculate product of list items """
    if l:
        return l.pop() * lprod(l)
    else:
        return 1

quote:

Let�s trace this function using this list:

[2, 2, 3, 1]

So we enter the function for the first time. List is populated, so we return l.pop(), which is 1, and lprod(l), which is lprod([2, 2, 3]). When the latter returns its value, it will be multiplied by the popped item.

Python hobbyist/newbie here. Can someone please explain to me why that works? How is he using lprod(l) as a value inside the function definition of it?

# ? Dec 24, 2010 03:34

A A 2 3 5 8 K: Nov 24, 2003; Illiteracy... what does that word even mean?

Swanson Broth posted:

code:
def lprod(l):
    """ Calculate product of list items """
    if l:
        return l.pop() * lprod(l)
    else:
        return 1
Python hobbyist/newbie here. Can someone please explain to me why that works? How is he using lprod(l) as a value inside the function definition of it?

Whenever lprod is called, it then calls itself with a list with one fewer element than it started with. Until it's called with an empty list, which ends the series of calls and returns an integer.

The first time lprod([2,2,3,1]) is called, it returns 1 * lprod([2,2,3]), which is just lprod([2,2,3]). Continuing:

lprod([2,2,3]) = 3 * lprod([2,2])
lprod([2,2]) = 2 * lprod([2])
lprod([2]) = 2 * lprod([])
lprod([]) = 1

So, working backwards, we have 1 * 2 * 2 * 3, or 12. It's called recursion, if you want to read more about it.

# ? Dec 24, 2010 03:59

Stabby McDamage: Dec 11, 2005; Doctor Rope

Swanson Broth posted:

code:
def lprod(l):
    """ Calculate product of list items """
    if l:
        return l.pop() * lprod(l)
    else:
        return 1
Python hobbyist/newbie here. Can someone please explain to me why that works? How is he using lprod(l) as a value inside the function definition of it?

That's a really bad way of implementing that, actually. Lists are mutable, which means that they are passed by reference and can be modified. So every time this thing calls itself to multiply the next number, it literally removes a value from the list you passed it. As in, it modifies what the caller passed, e.g.:

code:

>>> a=[1,2,3]
>>> a
[1, 2, 3]
>>> lprod(a)
6
>>> a
[]

This is pretty bad, since it destroys the list just to multiply it up! You could rewrite this function with the author's original intent without this flaw using slices:

code:

def lprod(l):
    """ Calculate product of list items """
    if l:
        return l[0] * lprod(l[1:])
    else:
        return 1

This is much less efficient, because it has to build a copy of most of the list when you do l[1:]. You could make an efficient recursive version if you passed an index around, but a function like this would almost always be written iteratively anyway, e.g.:

code:

def lprod(l):
    """ Calculate product of list items """
    r=1
    for v in l:
        r = r * v
    return r

In Python, though, you could just as easily use reduce:

code:

def lprod(lst): return reduce(lambda a,b:a*b, lst, 1)

# ? Dec 24, 2010 06:10

uncleTomOfFinland: May 25, 2008

I have this 3rd party ircbot/markov chain program thing written in Python and it's really thrashing the hard drive every time it generates replies to user messages. I have looked through the source code and can't figure out whats really causing it since it appears to me that the bot simply loads the database files into memory completely, marshall.loads() the huge string objects into dictionary objects and then goes on to do whatever with the dictionary objects. How should I proceed with trying to find out what is causing all this disk activity? Just pointing me at the right module/software would suffice I think.

# ? Dec 24, 2010 10:47

ErIog: Jul 11, 2001

Well, are the database files so large that your computer is swapping them out to disk then thrashing on access? If there's a significant number of items it's loading, that application might be just not built to handle large datasets efficiently. I mean, that sounds like a poor way for that program to operate. Something like sqlite would be a much better solution. It's pretty much built to do searching and indexing of on-disk objects efficiently.

Are these databases static or are they the input that you're providing?

ErIog fucked around with this message at 15:40 on Dec 24, 2010

# ? Dec 24, 2010 15:33

bc87: Dec 11, 2010

Can someone tell me why is variable a considered higher than variable b??

I have a very simple program meant to tell which variable is the maximum.

here's the code

code:

def printMax(a, b):
	if a > b:
		print "+++  the variable a =",a, 'is the maximum  +++'
	else:
		print "+++  the variable b=",b, 'is the maximum  +++'
	print "\n\n\n\n\n"
	
while True:
        print "\n"
        a = raw_input('***  What number to put for a?\n')
        if a == "quit":
                break
        print "\n"
        b = raw_input('what number to put for b?\n')
        if b == "quit":
                break
        print "\n"
        printMax(a,b)

Does this have to do with raw_input?

# ? Dec 25, 2010 01:29

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

234 > 1234 is false
"234" > "1234" is true

you need printMax(int(a), int(b))

# ? Dec 25, 2010 01:34

bc87: Dec 11, 2010

Janin posted:

234 > 1234 is false
"234" > "1234" is true

you need printMax(int(a), int(b))

I just tried it right now and I got this

The syntax is wrong, but I get that I have to make it an integer somehow.

bc87 fucked around with this message at 01:42 on Dec 25, 2010

# ? Dec 25, 2010 01:40

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

bc87 posted:

I just tried it right now and I got this

The syntax is wrong, but I get that I have to make it an integer somehow.

don't use int in the def for printMax.

Use it when you call printMax.

# ? Dec 25, 2010 01:41

bc87: Dec 11, 2010

Thermopyle posted:

don't use int in the def for printMax.

Use it when you call printMax.

Thanks, I got it working now. I'm learning python, and new to programming, that's why I don't get this poo poo at first.

# ? Dec 25, 2010 01:45

Jonnty: Aug 2, 2007; The enemy has become a flaming star!

bc87 posted:

Thanks, I got it working now. I'm learning python, and new to programming, that's why I don't get this poo poo at first.

It's cool, you'll get it eventually.

(you can say poo poo by the way)

# ? Dec 25, 2010 01:52

awesomepanda: Dec 26, 2005; The good life, as i concieve it, is a happy life.

For the function open() is there a way to specify the directory it will look into? i dont want to type in the whole address into the function.

awesomepanda fucked around with this message at 23:15 on Dec 26, 2010

# ? Dec 26, 2010 23:11

No Safe Word: Feb 26, 2005

CrazyPanda posted:

For the function open() is there a way to specify the directory it will look into? i dont want to type in the whole address into the function.

You can use os.chdir

# ? Dec 26, 2010 23:19

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

CrazyPanda posted:

For the function open() is there a way to specify the directory it will look into? i dont want to type in the whole address into the function.

You're using a programming language. Variables are your friend.

Not knowing exactly what your question means...try this...

code:

import os

def openfilesforscience(filename):
    directory = "/some/frickin/directory/"
    with open(os.path.join(directory, filename)) as whatupdudes:
       #do awesome stuff with this file

# ? Dec 26, 2010 23:20

awesomepanda: Dec 26, 2005; The good life, as i concieve it, is a happy life.

Hi thanks Thermopyle and No Safe Word.

What directory does open() look into when opening the file?

# ? Dec 26, 2010 23:25

No Safe Word: Feb 26, 2005

CrazyPanda posted:

Hi thanks Thermopyle and No Safe Word.

What directory does open() look into when opening the file?

The current working directory, which you can get via os.getcwd and change via os.chdir as I mentioned

# ? Dec 26, 2010 23:29

awesomepanda: Dec 26, 2005; The good life, as i concieve it, is a happy life.

No Safe Word posted:

The current working directory, which you can get via os.getcwd and change via os.chdir as I mentioned

Thank you so very much!

# ? Dec 26, 2010 23:34

maskenfreiheit: Dec 30, 2004

Kind of new to python... what would I want to use if I have an HTML document I wanna grab table data from? Is there an easy way via some pre-made method or will I have to get my hands dirty and regex?

# ? Dec 27, 2010 00:15

spankweasel: Jan 4, 2006

GregNorc posted:

Kind of new to python... what would I want to use if I have an HTML document I wanna grab table data from? Is there an easy way via some pre-made method or will I have to get my hands dirty and regex?

Ugh. Don't use regex to parse HTML. Ever.

lxml is your friend.

http://codespeak.net/lxml/

http://codespeak.net/lxml/parsing.html

# ? Dec 27, 2010 00:35

king_kilr: May 25, 2007

PyQuery! http://pypi.python.org/pypi/pyquery

# ? Dec 27, 2010 00:37

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:23 on Mar 13, 2017

# ? Dec 27, 2010 01:51

Adbot: ADBOT LOVES YOU

# ? Jun 3, 2024 04:38

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

GregNorc posted:

I guess I should clarify that I'm not looking to do anything with said html, just strip out the data in a column in a table...

Both of the suggestions provided are good for that.

# ? Dec 27, 2010 01:53

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »