Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

hey mom its 420: May 12, 2007

Yeah but in the case of except: pass, you're implicitly silencing a bunch of exceptions like KeyboardInterrupt and the like.

# ? Nov 15, 2008 18:31

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 00:31

tbradshaw: Jan 15, 2008; First one must nail at least two overdrive phrases and activate the tilt sensor to ROCK OUT!

chemosh6969 posted:

A python application for following the US primaries

edit: According to the Zen of Python, I'm right.

"Errors should never pass silently.
Unless explicitly silenced."
-- Tim Peters' Zen of Python

No, you are not explicitly silencing the error. You're just turning off error handling completely and implicitly silencing whatever error you're actually expecting. (Along with every other possible exception that you aren't expecting.)

Of the many pieces of advice you've been given on the subject, explicitly handling exceptions has been the most numerous.

e:f,b

# ? Nov 15, 2008 18:57

duck monster: Dec 15, 2004

king_kilr posted:

Guys, this: http://github.com/alex/pyelection/tree/master/models.py#L57 is why you shouldn't be using except: pass, because i don't have a loving clue what my own code does, or what the logic is.

In a busy workplace most python programers will probably confess to resorting to *occasionally* crash-mode boundary checking to get things done fast, namely something like

code:

try:
  x = y[z]
except:
  x = 0

That will pretty much always work, sorta, except its loving terrible, because there are unforseen things and those should NOT be allowed to work!. y might be some whacky iiteratating thing thats actually slurping off a database, and perhaps by smacking out the except your breaking some transaction handling. You are probably whacking keyboard handling (this is a big one when your dealing with a shitload of noisy data and you go 'gently caress it, just crash through the bad records , jobs gotta be out the door in 15 mins' and you run it and you cant even ^C out because you except:pass ed it.

Everyones been there, but it is considered BAD mojo.

What you should be doing is first testing bounds etc , and THEN access stuff. If you must just crash past something , then try and work out in advance what the error is, and learn how to use except to catch *specific* errors.

# ? Nov 15, 2008 19:38

duck monster: Dec 15, 2004

ah, its beautiful-soup based webpage scraping. Yeah ok, that poo poo can get very messy and I can relate to 'gently caress it we are going live' motherfucker-style exception handling.

# ? Nov 15, 2008 19:45

Lonely Wolf: Jan 20, 2003; Will hawk false idols for heaps and heaps of dough.

duck monster posted:

code:

try:
  x = y[z]
except:
  x = 0

code:

x = getattr(y, z, 0)

# ? Nov 15, 2008 19:55

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

Lord Uffenham posted:

code:

x = getattr(y, z, 0)

This won't work because z is not an attribute of y, you probably want this instead:

code:

x = y.get (z, 0)

which uses the dictionary interface

# ? Nov 15, 2008 19:59

tef: May 30, 2004; -> some l-system crap ->

If you're using beautiful soup still, try using lxml instead. You can use it to parse html and xml quickly, and run xpath over it.

# ? Nov 15, 2008 20:40

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

Janin posted:

This won't work because z is not an attribute of y, you probably want this instead:
code:
x = y.get (z, 0)
which uses the dictionary interface

Which, of course, won't work if it's a sequence instead of a mapping. You either have to catch the IndexError or do the bounds checking manually.

tef posted:

If you're using beautiful soup still, try using lxml instead. You can use it to parse html and xml quickly, and run xpath over it.

Sure, except lxml can segfault with exceptionally malformed HTML. BeautifulSoup might be slower and not support xpath, but it's pretty hard to get it to crash. That's why you'd use BeautifulSoup in the first place.

# ? Nov 15, 2008 20:41

king_kilr: May 25, 2007

tef posted:

If you're using beautiful soup still, try using lxml instead. You can use it to parse html and xml quickly, and run xpath over it.

I'm not talking about the BeautifulSoup stuff, I'm saying I have no idea what exceptions I'm trying to catch, or what circumstances lead to those things.

# ? Nov 15, 2008 21:17

Lazlo: May 1, 2002; Lately I've come to realize that I have certain materialistic needs.

I am pretty much a novice when it comes to programming, I have some experience with C++ from college, and a little knowledge of Java. I would like to write a program to help automate some tasks at work, specifically parsing out information of interest from our log files.

These log files are arranged in a hierarchical fashion, and I am interested in lines with the �[NUM]� tag, the descriptor, as well as its position within the hierarchy. The log files themselves aren't too large, between 10k and 40k in size. I want to be easily able to search the parsed results for specific [NUM] values, and display its hierarchical position. (The goal being to eliminate the current method used, a tedious task of doing repetitive searches in notepad).

I would be working in a Linux environment, and while I am not averse to learning a new language, I also want this to be something I could actually accomplish in my spare time. From what I've read so far, using regular expressions with Python would be a good match, in that it is well suited to the task and not too difficult to learn.

My question is, given the experience (or lack thereof) I have, and what I want to accomplish, would Python be the best choice? And if so, what tools might I need to get started?

Example of the log file contents I want to process:

code:

    (1.1)HAS SDE CONTEXT:Subject Class(LNX,13254)[TAG] = Person(LNX,121025)
    (1.2)HAS SDE CONTEXT:Subject Name(LNX,134945)[PNAME] = NAME
    (1.3)CONTAINS:Person Characteristics(LNX,1435934)[CONTAINER] = {SEPARATE}
        (1.3.1)CONTAINS:SubCatagory1(LN,189316-6)[NUM] = 3 no units
        (1.3.2)CONTAINS:SubCatagory2(LN,13177-6)[NUM] = 2 no units
    (1.4)CONTAINS:Summary(LNX,1216511)[CONTAINER] = {SEPARATE}
        (1.4.1)CONTAINS:CDF(LN,15255-2)[DATE] = 20070524
        (1.4.2)CONTAINS:Con Summary(LNX,134568)[CONTAINER] = {SEPARATE}
            (1.4.2.1)HAS SDE CONTEXT:Con ID(LN,4321-1)[TEXT] = A
            (1.4.2.2)CONTAINS:Con X(LN,457621-1)[NUM] = 151 day
            (1.4.2.3)CONTAINS:Composite Age(LN134346-5)[NUM] = 133 day
            (1.4.2.4)CONTAINS:LGT rank(LN,123647-1)[NUM] =
                (1.4.2.4.1)INFERRED FROM:Table of Values Citation(LNX,34567)[TAG]= XYZ

# ? Nov 16, 2008 02:03

JoeNotCharles: Mar 3, 2005; Yet beyond each tree there are only more trees.

Lazlo posted:

I would be working in a Linux environment, and while I am not averse to learning a new language, I also want this to be something I could actually accomplish in my spare time. From what I've read so far, using regular expressions with Python would be a good match, in that it is well suited to the task and not too difficult to learn.

My question is, given the experience (or lack thereof) I have, and what I want to accomplish, would Python be the best choice? And if so, what tools might I need to get started?

Python's excellent for this. You shouldn't need anything outside the basic Python package (almost certainly already installed with your Linux) and the docs on python.org.

The other common choice for parsing log files and things is Perl, but it's a lot harder to learn than Python because its syntax is less clean and well structured. The only benefit (for this task) is that it has a lot of shortcuts built in so once you get good with it you can write regular expression code faster (ie. with less typing).

# ? Nov 16, 2008 02:29

Lazlo: May 1, 2002; Lately I've come to realize that I have certain materialistic needs.

JoeNotCharles posted:

Python's excellent for this. You shouldn't need anything outside the basic Python package (almost certainly already installed with your Linux) and the docs on python.org.

The other common choice for parsing log files and things is Perl, but it's a lot harder to learn than Python because its syntax is less clean and well structured. The only benefit (for this task) is that it has a lot of shortcuts built in so once you get good with it you can write regular expression code faster (ie. with less typing).

Thanks, I'm looking through some of the tutorials and documentation. I do have another general question though. There are a lot of choices on editors, and debuggers and so forth, I'm currently jostling between emacs to edit, gdb to debug, and another window for the interpreter. Is this how it's done? It's ok for "Hello World", but it seems a little painful/awkward for anything more ambitious then that.

# ? Nov 16, 2008 03:37

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

Lazlo posted:

Thanks, I'm looking through some of the tutorials and documentation. I do have another general question though. There are a lot of choices on editors, and debuggers and so forth, I'm currently jostling between emacs to edit, gdb to debug, and another window for the interpreter. Is this how it's done? It's ok for "Hello World", but it seems a little painful/awkward for anything more ambitious then that.

gdb is definitely way overkill for python. If you need a debugger, there's one in the stdlib. You can use it by invoking your script as python -m pdb scriptname.py. I believe there's documentation on how to use it on http://docs.python.org/ as well.

If you're comfortable with emacs, it works just fine for editing python syntax. You can also run your script through emacs, but I've seen some weird quirks regarding that. I'd recommend using a separate window for invoking your script like you're doing now.

I don't really recommend using regexps in general. They're good for tokenizing text, but can be a pain for other things. pyparsing isn't in the stdlib, but it's a good package and there's quite a few examples of how to use it on their website. It wouldn't be difficult to build a pyparsing grammar to represent each line in the logfile.

# ? Nov 16, 2008 04:21

cowboy beepboop: Feb 24, 2001

I like SPE because the syntax highlighting is pretty

# ? Nov 16, 2008 04:33

tef: May 30, 2004; -> some l-system crap ->

king_kilr posted:

I'm not talking about the BeautifulSoup stuff.

It was merely an aside

- I found lxml nicer to use.

Habnabit posted:

Sure, except lxml can segfault with exceptionally malformed HTML.

Can you elaborate on this - we've migrated a scraper library from beautiful soup to lxml and it has been pretty solid so far, but it would not be good to try to run the transition in reverse

# ? Nov 16, 2008 04:44

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

tef posted:

Can you elaborate on this - we've migrated a scraper library from beautiful soup to lxml and it has been pretty solid so far, but it would not be good to try to run the transition in reverse

Unfortunately I don't know the details of this. It might've improved, but I remember some people having trouble with lxml doing this at least a few months ago on #python. lxml does have a mailing list, and there might be more useful information archived there.

e: Briefly looking over the mailing list archives, I think it might've been an OS X-specific issue. I really need to go to bed or I'd investigate further. Sorry.

Habnabit fucked around with this message at 11:24 on Nov 16, 2008

# ? Nov 16, 2008 11:22

The Remote Viewer: Jul 9, 2001

How long will it take me to get to the point where I can create a very basic roguelike in Python, with some minor past experience in programming?

# ? Nov 16, 2008 22:48

chemosh6969

Jul 3, 2004

code:
cat /dev/null > /etc/professionalism

_{_{I am in fact a massive asswagon.

Do not let me touch computer.}}

JoeNotCharles posted:

The other common choice for parsing log files and things is Perl

If someone's on linux, I'd use sed/grep/awk before using Perl and possibly python.

# ? Nov 16, 2008 23:44

bitprophet: Jul 22, 2004; Taco Defender

The Remote Viewer posted:

How long will it take me to get to the point where I can create a very basic roguelike in Python, with some minor past experience in programming?

It's kind of hard to give any sort of real estimate, it's like asking "how long till I get my black belt?" in a martial art -- it's different for everyone.

That said, Python should be quicker than almost any other language, given its syntax (easy and powerful) and the available libraries (core and third-party).

For example, I'm sure there are curses libraries out there (I think there's at least one in the stdlib) which would obviously take a lot of the work out of the actual UI components; depending on how you want to track state, there's a lot of modules for databases (RDBMS or small stuff like SQLite or bdb) and just general persistence (pickling, marshaling etc). And so forth.

Put another way, I'd bet your major hurdles will be understanding the overall concepts involved (screen output / redrawing, handling user input, data structures, keeping state, the concept of data persistence, and so on) and less about the specifics of the code itself.

# ? Nov 17, 2008 01:24

chemosh6969

Jul 3, 2004

code:
cat /dev/null > /etc/professionalism

_{_{I am in fact a massive asswagon.

Do not let me touch computer.}}

Is there a way to decompile or read a .pyd file?

# ? Nov 17, 2008 20:04

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

chemosh6969 posted:

Is there a way to decompile or read a .pyd file?

It's a standard DLL, any disassembler should work.

# ? Nov 17, 2008 21:40

The Remote Viewer: Jul 9, 2001

Learning programming is such mindless loving drudgery. Coding is fun, learning how to code sucks, especially self-learning. I've never learned how to do anything worthwhile in any language because I can't get past the hump where enthusiasm starts to flag. At least outside of what I learned in high school.

When you first start out there's a little bit of excitement from embarking on a new venture, and then I assume you eventually get to the point where you can start working on your own projects which boosts enthusiasm again. Being in the middle where you're retyping examples of number-guessing games, sucks.

# ? Nov 18, 2008 00:16

chemosh6969

Jul 3, 2004

code:
cat /dev/null > /etc/professionalism

_{_{I am in fact a massive asswagon.

Do not let me touch computer.}}

I get more inspired when I'm doing stuff that automates crap at work.

I'm wondering if there's a better way to download email so it removes all the '\n\r\n' stuff.

Here's what I'm doing

code:

from imaplib import *
server = IMAP4('server')
server.login('username', 'password')
r = server.select('INBOX')
r, data = server.search(None, '(FROM \'someone\')')
#let's say I get the 2293 from data
r, data = server.fetch('2293', '(BODY[TEXT])')

When I do "print data" or write to a file "outfile.write(str(data))", the output is always stuff like "Transaction Count: 3\r\n\r\n\r\nSource"

Is there something I can do to get rid of the new lines and stuff? Off the top of my head would be a substitution on \r and \n to \\r and \\n but there has to be a better way.

# ? Nov 18, 2008 01:11

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

chemosh6969 posted:

I get more inspired when I'm doing stuff that automates crap at work.

I'm wondering if there's a better way to download email so it removes all the '\n\r\n' stuff.

Here's what I'm doing
code:
from imaplib import *
server = IMAP4('server')
server.login('username', 'password')
r = server.select('INBOX')
r, data = server.search(None, '(FROM \'someone\')')
#let's say I get the 2293 from data
r, data = server.fetch('2293', '(BODY[TEXT])')
When I do "print data" or write to a file "outfile.write(str(data))", the output is always stuff like "Transaction Count: 3\r\n\r\n\r\nSource"

Is there something I can do to get rid of the new lines and stuff? Off the top of my head would be a substitution on \r and \n to \\r and \\n but there has to be a better way.

repr(data) would be an easy way to do it, or data.encode('string_escape') if you don't want the quotes around the whole thing. I don't know imaplib, so there might be a more elegant way to get the data you want, but that's the easiest way to escape escape characters and nonprintables.

# ? Nov 18, 2008 01:28

chemosh6969

Jul 3, 2004

code:
cat /dev/null > /etc/professionalism

_{_{I am in fact a massive asswagon.

Do not let me touch computer.}}

Habnabit posted:

repr(data) would be an easy way to do it, or data.encode('string_escape') if you don't want the quotes around the whole thing. I don't know imaplib, so there might be a more elegant way to get the data you want, but that's the easiest way to escape escape characters and nonprintables.

I'm tired and have no clue what I'm doing.

I guess my question should be, if I'm writing the string with \n, should it make a new line in the file? Instead it's showing all the \r\n characters in the file itself.

edit: I think I found the error. I need to string up the insides.

chemosh6969 fucked around with this message at 01:57 on Nov 18, 2008

# ? Nov 18, 2008 01:54

duck monster: Dec 15, 2004

The Remote Viewer posted:

How long will it take me to get to the point where I can create a very basic roguelike in Python, with some minor past experience in programming?

spend a shitload of time learning yourselves lists, tuples and dictionaries, and then try and get your OO mojo up.

Rogue-likes are probably right up pythons alley.

# ? Nov 18, 2008 01:56

deimos: Nov 30, 2006; Forget it man this bat is whack, it's got poobrain!

duck monster posted:

spend a shitload of time learning yourselves lists, tuples and dictionaries, and then try and get your OO mojo up.

Rogue-likes are probably right up pythons alley.

I am disappointed import rogue isn't part of the standard library.

# ? Nov 18, 2008 02:14

BlackDiamonds: Mar 15, 2007

For those of you who are playing around with python editors I want to say that the dev branch for Python in NetBeans is fantastic. See http://wiki.netbeans.org/Python and http://blogs.sun.com/tor/ for more information. I'm just sad that it won't be in NetBeans 6.5.

# ? Nov 18, 2008 23:24

nonathlon: Jul 9, 2004; And yet, somehow, now it's my fault ...

BlackDiamonds posted:

For those of you who are playing around with python editors I want to say that the dev branch for Python in NetBeans is fantastic. See http://wiki.netbeans.org/Python and http://blogs.sun.com/tor/ for more information. I'm just sad that it won't be in NetBeans 6.5.

Colour me disappointed. Netbeans is my editor of choice - I work in a number of languages and prefer to use the one editor for all - and the lack of Python/Jython support has chaffed. Have you been using nbpython with the 6.5 candidates?

# ? Nov 18, 2008 23:37

cowboy beepboop: Feb 24, 2001

Welp, I just compiled and installed Python 2.6 on Ubuntu 8.10 and it broke all my Python apps. Just a word of warning for everyone

# ? Nov 19, 2008 03:02

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

my stepdads beer posted:

Welp, I just compiled and installed Python 2.6 on Ubuntu 8.10 and it broke all my Python apps. Just a word of warning for everyone

I assume if you run "python" from a terminal, it shows this (or something similar):

code:

Python [b]2.6[/b] (r26:66714, Nov  18 2008, 16:40:21) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

If it does, run "which python". If it comes up "/usr/local/bin/python", then just remove that file and it should work fine. If it prints "/usr/bin/python", then run "ln -sf python2.5 /usr/bin/python".

though if you installed to /usr, you've possibly screwed up a good portion of your system.

# ? Nov 19, 2008 04:05

MononcQc: May 29, 2007

Janin posted:

I assume if you run "python" from a terminal, it shows this (or something similar):
code:
Python [b]2.6[/b] (r26:66714, Nov  18 2008, 16:40:21) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
If it does, run "which python". If it comes up "/usr/local/bin/python", then just remove that file and it should work fine. If it prints "/usr/bin/python", then run "ln -sf python2.5 /usr/bin/python".

though if you installed to /usr, you've possibly screwed up a good portion of your system.

I just downloaded the files in my home folder, installed it there and made a symlink to the binary file in /usr/bin/python2.6. I don't know if there's much trouble related to this, but I can run my stuff without a problem that way. I have yet to install a 2.6-exclusive module though.

# ? Nov 19, 2008 04:22

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

MononcQc posted:

I just downloaded the files in my home folder, installed it there and made a symlink to the binary file in /usr/bin/python2.6. I don't know if there's much trouble related to this, but I can run my stuff without a problem that way. I have yet to install a 2.6-exclusive module though.

This is the safe way to do it, I'm guessing beer hosed up and installed it to /usr because all the Python apps I've looked at use /usr/bin/python for their path.

# ? Nov 19, 2008 05:00

JoeNotCharles: Mar 3, 2005; Yet beyond each tree there are only more trees.

They should be using "/usr/bin/env python" so they don't break when you run them on a different system.

# ? Nov 19, 2008 06:16

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

JoeNotCharles posted:

They should be using "/usr/bin/env python" so they don't break when you run them on a different system.

What systems don't install Python in /usr/bin, or at least provide a symlink? Using /usr/bin/env is a risk in scripts because it relies on the user's PATH to be set correctly; if it's broken or compromised, the script will not work correctly / at all.

If you need to run a script using the version of Python on your path but not in /usr/bin, then use "python /path/to/myscript".

Also, using /usr/bin/env doesn't allow additional parameters like -tt to be passed to the interpreter.

# ? Nov 19, 2008 06:41

tbradshaw: Jan 15, 2008; First one must nail at least two overdrive phrases and activate the tilt sensor to ROCK OUT!

Janin posted:

What systems don't install Python in /usr/bin, or at least provide a symlink?

Those systems that have Python in /usr/local/bin.

Those systems with Python in /opt/python

Those systems that don't have python natively at all, but Python has been installed in the home directory of the user.

Many systems are configured in "nonstandard" ways. Maybe the simple question is "What systems can run Python when it is not installed by root?"

All of them.

Most importantly, the issue is not whether /usr/bin/python exists, it's whether you're supposed to be using it! I have Python compiled on many hosts that have an older version of Python on the system, but I have the version I want to use in ~/local/. Use "#!/usr/bin/env python" because it's not appropriate for you (as a developer) to decide that "users are stupid and will mess up their path, so I will force my code to use the system Python". "$HOME/local/bin/" comes before "/usr/bin" in my PATH environment variable for a reason!

tbradshaw fucked around with this message at 07:15 on Nov 19, 2008

# ? Nov 19, 2008 07:11

JoeNotCharles: Mar 3, 2005; Yet beyond each tree there are only more trees.

Janin posted:

Also, using /usr/bin/env doesn't allow additional parameters like -tt to be passed to the interpreter.

Sure it does. It passes all parameters through.

# ? Nov 19, 2008 07:51

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

If you think Python scripts should use #!/usr/bin/env python, do you also code your shell scripts to use #!/usr/bin/env sh?

JoeNotCharles posted:

Sure it does. It passes all parameters through.

code:

[~]$ ./test.sh 
/usr/bin/env: python -tt: No such file or directory

# ? Nov 19, 2008 08:03

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

Janin posted:

If you think Python scripts should use #!/usr/bin/env python, do you also code your shell scripts to use #!/usr/bin/env sh?
code:
[~]$ ./test.sh 
/usr/bin/env: python -tt: No such file or directory

This actually depends on your kernel.

# ? Nov 19, 2008 08:30

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 00:31

bitprophet: Jul 22, 2004; Taco Defender

Janin posted:

If you think Python scripts should use #!/usr/bin/env python, do you also code your shell scripts to use #!/usr/bin/env sh?

That's not a fair comparison; any system ever is going to have some shell binary symlinked/installed as /bin/sh, but as others have already explained it's common for Python to be installed in any of a multitude of different locations, not just /usr/bin/python.

# ? Nov 19, 2008 16:07

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »