Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

NadaTooma: Aug 24, 2004; The good thing is that everyone around you has more critical failures in combat, the bad thing is - so do you!

HatfulOfHollow posted:

I'm pretty sure this replaces that entire monstrosity...

I was bored, so I took a shot at it too. Seems to work well enough from my quick testing...

code:

#!/usr/bin/env python

import os.path, shutil

# Build all our file names.
base_path = "c:/Documents and Settings/Name/Desktop/HTMLiser"
letters   = "ABCDEFGHIJ"

# Read in all our replacement data files.
def load_input_file(fname):
    global base_path
    full_name = os.path.join(base_path, fname)
    lines     = open(full_name, "rt").readlines()
    return [x.strip() for x in lines]

titles     = load_input_file("Input/Titles.txt")
paragraphs = load_input_file("Input/Paragraphs.txt")
tracer     = load_input_file("Input/TracerImages.txt")
reff_no    = load_input_file("Input/ReffNo.txt")

# Generate the ten output files from our template, using our replacements.
template_fname = os.path.join(base_path, "Input/Template.txt")
for index in range(len(letters)):
    replacements = [
        ("TITLE",           titles[index]),
        ("PARAGRAPH",       paragraphs[index]),
        ("TRACERIMAGE",     tracer[index]),
        ("REFERENCENUMBER", reff_no[index])]
    text = open(template_fname, "rt").read()
    for (tag, value) in replacements:
        text = text.replace("@@@@@%s@@@@@" % tag, value)
    output_fname  = os.path.join(base_path, "Output/12-000211-%s-Zuneta1.htm" % letters[index])
    open(output_fname, "wt").write(text)

# ? Aug 12, 2009 22:18

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 14:30

TheMaskedUgly: Sep 21, 2008; Let's play a different game.

You people and your making me look bad. I'll have a look over those, and if they work, I love you. Even if they don't I appreciate the effort. Thanks, really.
Any ideas why it stopped before the end, or should I just chalk that up to I suck at python and it got angry at that?

E: Just checked, both are perfect; I'm indebted to you.

TheMaskedUgly fucked around with this message at 00:42 on Aug 13, 2009

# ? Aug 12, 2009 23:59

Casey Finnigan: Apr 30, 2009; Dumb ✔
So goddamn crazy ✔

Lonely Wolf posted:

It looks like it's running fine except you didn't pass it any command line arguments and you're running it on windows instead of a *nix. There does not appear to be a problem. What do you think is wrong?

Try running it from the command line like
code:
python using_sys.py We are arguments

Yeah it seems you're right. It wasn't printing 'We are arguments' when I ran or imported it so I figured that I just totally hosed everything up.

Since that dumb question is dealt with, I have another (possibly more stupid. I'm pretty poo poo with computers):
In this lesson the guy codes a basic backup program and uses his OS's built-in 'zip' function to zip the files. On the previous page he recommends getting Info-Zip for Windows users. I tried that and ZLib but I'm totally lost when I try to use them with Python. If anyone could steer my dumb rear end on track it'd be appreciated.

# ? Aug 13, 2009 01:08

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

TheMaskedUgly posted:

You people and your making me look bad. I'll have a look over those, and if they work, I love you. Even if they don't I appreciate the effort. Thanks, really.
Any ideas why it stopped before the end, or should I just chalk that up to I suck at python and it got angry at that?

E: Just checked, both are perfect; I'm indebted to you.

That's actually surprising. I didn't even test mine. I wrote it in notepad after tracing the original script in my head. I'm a badass.

# ? Aug 13, 2009 01:15

tripwire: Nov 19, 2004; _{ghost flow}

TheMaskedUgly posted:

You people and your making me look bad. I'll have a look over those, and if they work, I love you. Even if they don't I appreciate the effort. Thanks, really.
Any ideas why it stopped before the end, or should I just chalk that up to I suck at python and it got angry at that?

E: Just checked, both are perfect; I'm indebted to you.

I didn't bother to check but if you want to, check out the python debugger. It's included with python and you can see instructions on how to use it here: http://docs.python.org/library/pdb.html

Try setting a breakpoint before where it breaks and stepping through the code line by line to see whats up.

Alternatively, since you can run python interactively from a shell, try doing it that way and take advantage of the read-eval-print loop to check the values of variables and so on at your leisure.

# ? Aug 13, 2009 02:21

NadaTooma: Aug 24, 2004; The good thing is that everyone around you has more critical failures in combat, the bad thing is - so do you!

HatfulOfHollow posted:

That's actually surprising. I didn't even test mine. I wrote it in notepad after tracing the original script in my head. I'm a badass.

Yeah but mine calls strip() so it doesn't embed extra newlines. :v:

# ? Aug 13, 2009 15:17

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

NadaTooma posted:

Yeah but mine calls strip() so it doesn't embed extra newlines.

creating a working script without testing > using strip()

# ? Aug 13, 2009 15:26

compuserved: Mar 20, 2006; Nap Ghost

tripwire posted:

I didn't bother to check but if you want to, check out the python debugger. It's included with python and you can see instructions on how to use it here: http://docs.python.org/library/pdb.html

alternatively winpdb is a good multi-platform python debugger. i use it all the time.

http://www.winpdb.org/cgi-bin/moin.cgi/WinpdbTutorial

compuserved fucked around with this message at 19:24 on Aug 13, 2009

# ? Aug 13, 2009 19:21

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

Threep posted:

I'm working on something that involves downloading a queue from a small number of different connections (say, 5 or so) to different files

What's the best way to go about this? I've looked at:

Python threads - because I doubt the GIL will hurt much when I'm mostly bandwidth-bound anyway.

multiprocessing module's queues - I'm not sure the best way to go about detecting a finished job though

And for the sake of learning - if I was downloading from 50/100 connections and/or to one file, what would the answer be?

Twisted! Anything else is just dumb. (Short of reimplementing twisted yourself, which is dumb for other reasons.) Threading would just add a lot of overhead: code complexity overhead and CPU overhead both.

# ? Aug 13, 2009 19:37

deedee megadoodoo: Sep 28, 2000; Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.

Habnabit posted:

Twisted! Anything else is just dumb. (Short of reimplementing twisted yourself, which is dumb for other reasons.) Threading would just add a lot of overhead: code complexity overhead and CPU overhead both.

I can write a multi-threaded application with no problem but for some reason twisted just confuses the hell out of me. I think it's the layer of abstraction. Not seeing exactly what's going on behind the scenes makes it harder for me to understand.

# ? Aug 13, 2009 19:46

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

HatfulOfHollow posted:

I can write a multi-threaded application with no problem but for some reason twisted just confuses the hell out of me. I think it's the layer of abstraction. Not seeing exactly what's going on behind the scenes makes it harder for me to understand.

I've noticed the same. It makes me feel dumb.

# ? Aug 13, 2009 21:23

king_kilr: May 25, 2007

HatfulOfHollow posted:

I can write a multi-threaded application with no problem but for some reason twisted just confuses the hell out of me. I think it's the layer of abstraction. Not seeing exactly what's going on behind the scenes makes it harder for me to understand.

No, twisted really is just that confusing. Also camelCase, and old style classes, and like 3 other things they do just piss me off.

# ? Aug 13, 2009 21:51

TheMaskedUgly: Sep 21, 2008; Let's play a different game.

HatfulOfHollow posted:

creating a working script without testing > using strip()

Hate to burst your bubble, but I did have to rename some of the classes, capitals in wrong places and whatnot. Come on man, you can do better than that. I kid, that's almost certainly my random capitalisation-ing.

# ? Aug 13, 2009 23:22

tripwire: Nov 19, 2004; _{ghost flow}

I hope you learned a valuable lesson about how not to code

Did you end up giving pdb or winpdb a shot?

# ? Aug 14, 2009 00:23

tef: May 30, 2004; -> some l-system crap ->

http://eventlet.net/ might be an alternative to twisted

# ? Aug 14, 2009 00:25

m0nk3yz: Mar 13, 2002; Behold the power of cheese!

Habnabit posted:

Twisted! Anything else is just dumb. (Short of reimplementing twisted yourself, which is dumb for other reasons.) Threading would just add a lot of overhead: code complexity overhead and CPU overhead both.

Yes. Threading adds so much complexity which twisted totally avoids because it's not complex at all. /sarcasm

# ? Aug 14, 2009 01:10

king_kilr: May 25, 2007

It's PyCon time again folks (sort of)! The call for proposals has just gone out, for any goons interested in sharing something with the Python community this is your big chance: http://pycon.blogspot.com/2009/08/pycon-2010-call-for-proposals.html

# ? Aug 14, 2009 01:12

Habnabit: Dec 30, 2007; lift your skinny fists like
antennas in germany.

m0nk3yz posted:

Yes. Threading adds so much complexity which twisted totally avoids because it's not complex at all. /sarcasm

At least problems in my twisted code are deterministic.

And really, twisted isn't as complicated as everyone makes it out to be. There's a lot of layers of abstraction, but relatively few that you actually have to touch yourself. Look over the finger tutorial sometime; it's pretty simple.

# ? Aug 14, 2009 06:28

m0nk3yz: Mar 13, 2002; Behold the power of cheese!

Habnabit posted:

At least problems in my twisted code are deterministic.

And really, twisted isn't as complicated as everyone makes it out to be. There's a lot of layers of abstraction, but relatively few that you actually have to touch yourself. Look over the finger tutorial sometime; it's pretty simple.

I'm pretty familiar with threading, multiprocessing, and twisted. Yes; it is complex in odd ways, and while some people think asynchronous programming is the One True Way, personally, I'm not sold on it. I've not had non-determinism problems using threading, and plenty of other people have been pretty successful using it.

# ? Aug 14, 2009 13:48

fart simpson: Jul 2, 2005; DEATH TO AMERICA

Speaking of twisted, does anyone have any idea why I can't get it to work anymore? Their latest release just won't work for me at all. I can import twisted, but anything more than that results in errors like "No module found names twisted.words". I've tried using the binary windows installer and python setup.py install on two different machines on WinXP SP2, SP3, and the Win7 RC, in Python 2.5, 2.6, stackless 2.5, and stackless 2.6. I'm getting by without it for now, but I'd really like it eventually! Has anyone encountered a similar problem?

# ? Aug 14, 2009 20:54

dizzywhip: Dec 23, 2005

So I've finally gotten around to starting to teach myself game development, and I'm doing so in Python with pyglet. I'm currently working on making a simple 2D platformer for the sake of learning. I'm trying to organize my code properly so that I can keep building on top of it without it turning into a giant mess. I have almost no experience working with larger projects, and I'm sure I'm going about it in a terrible and possibly offensive way.

It's organized so that I have a main game file, which imports all my classes and functions from various other files. I'm running into a problem with this, though. My Player class needs to know about the size of the game window for the purpose of collision detection, but it doesn't know about the window object since it's declared in the main game file.

I have a feeling I'm doing something fundamentally wrong here. What's the proper way to go about this situation?

# ? Aug 18, 2009 06:04

Scaevolus: Apr 16, 2007

mit_senf posted:

Pass along the world/game window object to Player in the constructor?

# ? Aug 18, 2009 11:15

dizzywhip: Dec 23, 2005

I had thought of that, but I figured there would be a better solution since it seems like there will be a lot of situations where I'd need to do this, and I think it's kind of ugly. But if that's the way to do it, then I'll do it. Thanks.

Edit: On that note, as a newbie to Python, I'm not sure how to keep a pointer to the window after passing it to the player in the constructor. If I set the window as an attribute of the Player object, it'll create a new copy of the window rather than a pointer, right?

dizzywhip fucked around with this message at 09:14 on Aug 19, 2009

# ? Aug 19, 2009 08:17

Lurchington: Jan 2, 2003; Forums Dragoon

I had a similar question earlier in this thread, and I ended up instantiating the primary object and importing it into the necessary file(s)

outlier posted:

Note that there's nothing especially Pythonic about this problem, it's a common design issue. There's a few ways you can go around this:

1. The central command class or function gathers the options and distributes them to more peripheral classes. You parse the options and then call "MyAppModel.set_options", which calls "DisplayController. set_options" and "DataReader. set_options" which may in turn call "DataParser. set_options", etc.

Advantages: centralised control of options, no part of the program can see or effect options that don't concern it.

Disadvantages: If you change or expand options, you have to walk your changes down the chain. And what if two parts of a program share options? Probably works better with smaller programs.

2. Have some sort of global settings / options object or class. See Django's settings and conf machinery for an example.

Advantages: easy. Everything can get at everything. Scales well.

Disadvantages: ohmigod, don't you now that globals are bad! Everything can get at everything!

I'd tip my hat towards the second for most cases, especially as a program grows in size.

Lurchington fucked around with this message at 14:50 on Aug 19, 2009

# ? Aug 19, 2009 14:46

checkeredshawn: Jul 16, 2007

Does anyone know how to retrieve a line immediately before a matched line with the re module? For example, if I have the text:

one
two
three
two

and I have the regular expression

p = re.compile('two')

can I get the lines containing the words "one" and "three" that are immediately before the matched "two"?

Thanks

Edit: Thank you vvvvvvvvvvvvvvvvvvvvvvv

checkeredshawn fucked around with this message at 15:14 on Aug 20, 2009

# ? Aug 19, 2009 19:24

tripwire: Nov 19, 2004; _{ghost flow}

code:

for index, line in enumerate(a_list_full_of_lines):
    my_match = p.match(line)
    if index > 0 and my_match:
        previous_line = a_list_full_of_lines[index-1]

# ? Aug 19, 2009 20:38

unixbeard: Dec 29, 2004

I am having a problem with unicode and webpages.

I'm pulling some data from an rss feed (the source), and putting it in another blog (wordpress) effectively making an rss aggregator (the target)

When i get the data from the rss, it looks like this:

haven�t

When i use the wordpress xml-rpc api to store the data in the target, it gets stored on the db like this:

haven\xe2\u20ac\u2122t

However, the xml-rpc api doesn't do what i want, so i'm manually inserting the data into the posts table, and where it is stored like this:

haven\u2019t

When i view it in the browser, the data uploaded via xml-rpc displays correctly, but the manually inserted data doesn't, giving me that unicode-wtf character.

I'm obviously missing some voodoo somewhere but i don't know where. I have checked both the mysql server and client are using utf-8 and the webpage character set is utf-8, any ideas?

# ? Aug 21, 2009 21:12

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

Wordpress is behaving as if the database is in windows-1252 encoding, instead of UTF-8. Double-check which encoding the tables were created with.

>>> u"haven�t".encode ("utf-8").decode("windows-1252") u'haven\xe2\u20ac\u2122t'

# ? Aug 22, 2009 02:45

chemosh6969

Jul 3, 2004

code:
cat /dev/null > /etc/professionalism

_{_{I am in fact a massive asswagon.

Do not let me touch computer.}}

I've never done web scrapping before and was wondering if anyone knew of, or had, an example of an imdb scrape? I want to learn how to do it for a few other sites but really wanted to find an example to build off of.

# ? Aug 22, 2009 03:45

nbv4: Aug 21, 2002; by Duchess Gummybuns

I have a set of variables like so:

code:

rly_important = ""
kinda_important = "t00t"
less_important = ""
not_important = "blah"

I want to set another variable as equal to the most important variable thats not blank. Is there a nice pythonic way of doing this, or will I have to use nested loops?

# ? Aug 22, 2009 07:40

dizzywhip: Dec 23, 2005

Why not make an array to store the variables from most to least important, then loop through it until you find the first non-blank one?

# ? Aug 22, 2009 08:21

nonathlon: Jul 9, 2004; And yet, somehow, now it's my fault ...

nbv4 posted:

I have a set of variables like so:
code:
rly_important = ""
kinda_important = "t00t"
less_important = ""
not_important = "blah"
I want to set another variable as equal to the most important variable thats not blank. Is there a nice pythonic way of doing this, or will I have to use nested loops?

code:

new_var = ""
for x in [rly_important, kinda_important, less_important, not_important]:
   if x:
      new_var = x
      break

Seems a pretty simple problem. Or is there some aspect I haven't picked up on?

# ? Aug 22, 2009 10:30

DICTATOR OF FUNK: Nov 6, 2007; aaaaaw yeeeeeah

chemosh6969 posted:

I've never done web scrapping before and was wondering if anyone knew of, or had, an example of an imdb scrape? I want to learn how to do it for a few other sites but really wanted to find an example to build off of.

Take a look at BeautifulSoup. It has an extensive documentation and would make the task loads easier.

This guy has a pretty good guide for how to use it.

# ? Aug 22, 2009 12:23

nbv4: Aug 21, 2002; by Duchess Gummybuns

outlier posted:

code:
new_var = ""
for x in [rly_important, kinda_important, less_important, not_important]:
   if x:
      new_var = x
      break
Seems a pretty simple problem. Or is there some aspect I haven't picked up on?

In python theres all sorts of crazy things you can do with things like [x.keys() for item in enumerate(list)] and lambdas and stuff. I thought there might be a better more idomatic way to do it without loops. But I guess you method works fine.

# ? Aug 22, 2009 18:14

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

nbv4 posted:

In python theres all sorts of crazy things you can do with things like [x.keys() for item in enumerate(list)] and lambdas and stuff. I thought there might be a better more idomatic way to do it without loops. But I guess you method works fine.

All of Python's fancy iteration functions are implemented with loops anyway. This case could be generalized to a useful function:

code:

def first(vs):
    for v in vs:
        if v:
            return v

new_var = first([rly_important, kinda_important, less_important, not_important])

# ? Aug 22, 2009 18:28

hlfrk414: Dec 31, 2008

You can abuse variable arguments if you want:

code:

def first(*vs):
    for v in vs:
        if v:
            return v

new_var = first(rly_important, kinda_important, less_important, not_important)

Or Take advantage that the boolean 'or' returns the first true value, which for strs is the first non-empty string. This will only work if you know the variables beforehand:

code:

non_empty = rly_important or kinda_important or less_important or not_important

It will return the first true value or the last false value, which would be the empty string. You can take advantage of this by sticking a default value at the end.

code:

non_empty = rly_important or kinda_important or less_important or not_important or 'all were empty'

Try not to abuse this, because it gets confusing, especially if you mix types.

# ? Aug 22, 2009 23:50

unixbeard: Dec 29, 2004

Janin posted:

Wordpress is behaving as if the database is in windows-1252 encoding, instead of UTF-8. Double-check which encoding the tables were created with.

>>> u"haven�t".encode ("utf-8").decode("windows-1252") u'haven\xe2\u20ac\u2122t'

hey thanks, i changed my mysql connection in python to use latin1, and run the rss content through .ecode("utf-8") and now everything seems to be working ok.

i was lookng at things with show create ...

the database:
otntest | CREATE DATABASE `otntest` /*!40100 DEFAULT CHARACTER SET utf8 */
the table:
CREATE TABLE `wp_4xgtmw_posts` ( `ID` bigint(20) unsigned NOT NULL auto_increment, ... ) ENGINE=MyISAM AUTO_INCREMENT=64 DEFAULT CHARSET=utf8 |

and if i look in phpmyadmin the columns of that table are using utf8_general_ci collation.

however i do see this on the server:

mysql> SHOW VARIABLES LIKE 'char%'; +--------------------------+-----------------------------------------+ | Variable_name | Value | +--------------------------+-----------------------------------------+ | character_set_client | latin1 | | character_set_connection | latin1 | | character_set_database | utf8 | | character_set_filesystem | binary | | character_set_results | latin1 | | character_set_server | latin1 | | character_set_system | utf8 | +--------------------------+-----------------------------------------+ 7 rows in set (0.00 sec) mysql>

it's working now which is cool, but i don't understand why i need to run the rss data through .encode() (isn't it already utf8 coming from the feed?) and why i need to set the db connection to use latin1 in python.

# ? Aug 23, 2009 17:26

TOO SCSI FOR MY CAT: Oct 12, 2008; this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"

unixbeard posted:

it's working now which is cool, but i don't understand why i need to run the rss data through .encode() (isn't it already utf8 coming from the feed?) and why i need to set the db connection to use latin1 in python.

You need to do both of these because Wordpress is configured incorrectly. If it used UTF-8 also, you wouldn't be running into these problems.

# ? Aug 23, 2009 17:59

chemosh6969

Jul 3, 2004

code:
cat /dev/null > /etc/professionalism

_{_{I am in fact a massive asswagon.

Do not let me touch computer.}}

root beer posted:

Take a look at BeautifulSoup. It has an extensive documentation and would make the task loads easier.

This guy has a pretty good guide for how to use it.

Thanks!!!

# ? Aug 24, 2009 00:44

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 14:30

unixbeard: Dec 29, 2004

Janin posted:

You need to do both of these because Wordpress is configured incorrectly. If it used UTF-8 also, you wouldn't be running into these problems.

ah ok, yeah it's the dreamhost default install (lol), i had a look and it doesn't define a character set to use, so must be defaulting to windwows-1252.

thanks for all your help

# ? Aug 24, 2009 07:05

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »