Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Lurchington
Jan 2, 2003

Forums Dragoon

hlfrk414 posted:

Use shelve?

I used shelve in a personal project and it worked fine. I'm a sucker for anything that is used like a dictionary.

edit:

king_kilr posted:

I used shelve for a personal project and I spent a lot of time cursing. It's not advertised but len(shelve) is O(n) and pull all of the objects into memory and doesn't dealloc then. Not a big deal if you're using shelve for persistance, but it means shelve is poor if you have a dataset who's size is greater than available RAM.

My data sets were pretty small, so I never risked the situation you're talking about.

VVVV

Lurchington fucked around with this message at 20:47 on Oct 24, 2009

Adbot
ADBOT LOVES YOU

king_kilr
May 25, 2007
I used shelve for a personal project and I spent a lot of time cursing. It's not advertised but len(shelve) is O(n) and pull all of the objects into memory and doesn't dealloc then. Not a big deal if you're using shelve for persistance, but it means shelve is poor if you have a dataset who's size is greater than available RAM.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

code:
>>> d['show1'] = show
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python26\lib\shelve.py", line 132, in __setitem__
    p.dump(value)
cPickle.UnpickleableError: Cannot pickle <type '_sre.SRE_Match'> objects
>>>
So what do I do if my object also includes regex.match objects?

tripwire
Nov 19, 2004

        ghost flow
What are you using the matches for? Extract the information or functionality you need from the matches, storing them in some other kind of format, and pickle that instead.

GuyGizmo
Feb 26, 2003

stupid babies need the most attention
This very well may have been answered in this thread already but can anyone recommend a good IDE for Python in Mac OS X?

Habnabit
Dec 30, 2007

lift your skinny fists like
antennas in germany.

GuyGizmo posted:

This very well may have been answered in this thread already but can anyone recommend a good IDE for Python in Mac OS X?

Textmate is pretty great, with a lot of different themes and a ridiculous number of syntax colorizers. If you don't want to pay money, TextWrangler is also pretty good.

Lurchington
Jan 2, 2003

Forums Dragoon

GuyGizmo posted:

This very well may have been answered in this thread already but can anyone recommend a good IDE for Python in Mac OS X?

Here's a link to the suggestions from when I asked it.

Honestly, I'm lead-developing on windows again just thanks to PyScripter

GuyGizmo
Feb 26, 2003

stupid babies need the most attention

Lurchington posted:

Here's a link to the suggestions from when I asked it.

Honestly, I'm lead-developing on windows again just thanks to PyScripter
Great, one of those earlier suggestions will surely work for me.

tripwire
Nov 19, 2004

        ghost flow
Hah! I thought this looked funny. I sometimes found myself writing a flattening function that would take any nested sequence and flatten it into a list. It would work fine except that I couldn't figure out how to flatten to an arbitrary depth without the ability to recurse to arbitrary depths. Well I saw this code posted by a guy called Danny Yoo for how to do this using continuation passing style.
code:
def flatten(a):
    """Flatten a list."""
    return bounce(flatten_k(a, lambda x: x))


def bounce(thing):
    """Bounce the 'thing' until it stops being a callable."""
    while callable(thing):
        thing = thing()
    return thing


def flatten_k(a, k):
    """CPS/trampolined version of the flatten function.  The original
    function, before the CPS transform, looked like this:

    def flatten(a):
        if not isinstance(a,(tuple,list)): return [a]
        if len(a)==0: return []
        return flatten(a[0])+flatten(a[1:])

    The following code is not meant for human consumption.
    """
    if not isinstance(a,(tuple,list)):
        return lambda: k([a])
    if len(a)==0:
        return lambda: k([])
    def k1(v1):
        def k2(v2):
            return lambda: k(v1 + v2)
        return lambda: flatten_k(a[1:], k2)
    return lambda: flatten_k(a[0], k1)
Has anyone heard of "trampoline" style programming before, to avoid overflowing the stack? It seems like such an evil concept :)

king_kilr
May 25, 2007

tripwire posted:

Has anyone heard of "trampoline" style programming before, to avoid overflowing the stack? It seems like such an evil concept :)

James Tauber has! http://jtauber.com/blog/2008/03/30/thunks,_trampolines_and_continuation_passing/ (disclaimer: he's my boss)

tef
May 30, 2004

-> some l-system crap ->

tripwire posted:

Has anyone heard of "trampoline" style programming before, to avoid overflowing the stack? It seems like such an evil concept :)

Sure, but it's easier to keep an explicit stack sometimes:

code:
$ cat flatten.py 
def flatten(input):
    output = []
    stack = []
    stack.extend(reversed(input))
    while stack:
        top = stack.pop()
        if isinstance(top, (list, tuple)):
            stack.extend(reversed(top))
        else:
            output.append(top)

    return output

print flatten([1,[2,3,[4,5,[6],7],8,[9,[10,11],[12],[[13]],[[[14,15]]],16]]])

$ python flatten.py 
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]

SlightlyMadman
Jan 14, 2005

I have a specialized url shortener Django app that works on base64 translations of a database ID. My boss is concerned that it could possibly generate offensive words as the base64 keys, and to be safe would like me to skip any ID whose base64 representation is a dictionary word.

I've found a couple of random aspell libraries for python, but they don't seem very well maintained. Is there a simpler way I can just check if a string is a word in the dictionary? My app runs on a linux (Ubuntu) server.

tef
May 30, 2004

-> some l-system crap ->
Looking around I found this: http://rightfootin.blogspot.com/2006/09/more-on-python-flatten.html

And it claimed a rather odd method was fastest,So I decided to run it against my own: http://pastebin.ca/1649364 is the test code.

Mine:
$ time python flattentest.py flatten 1000000
1000000

real 0m5.209s
user 0m4.934s
sys 0m0.145s



The one that 'whoops all of the others for speed, nesting level support and elegance':
$ time python flattentest.py flatten2 1000000
1000000

real 10m32.296s
user 9m55.045s
sys 0m4.894s


:c00l:

Edit: and the trampoline doesn't perform well either

tef fucked around with this message at 17:10 on Oct 30, 2009

A A 2 3 5 8 K
Nov 24, 2003
Illiteracy... what does that word even mean?

SlightlyMadman posted:

I've found a couple of random aspell libraries for python, but they don't seem very well maintained. Is there a simpler way I can just check if a string is a word in the dictionary? My app runs on a linux (Ubuntu) server.

dictionary = ['word1','word2','word3']

if word.lower() in dictionary:
# try again

tripwire
Nov 19, 2004

        ghost flow

tef posted:

Looking around I found this: http://rightfootin.blogspot.com/2006/09/more-on-python-flatten.html

And it claimed a rather odd method was fastest,So I decided to run it against my own: http://pastebin.ca/1649364 is the test code.

Mine:
$ time python flattentest.py flatten 1000000
1000000

real 0m5.209s
user 0m4.934s
sys 0m0.145s



The one that 'whoops all of the others for speed, nesting level support and elegance':
$ time python flattentest.py flatten2 1000000
1000000

real 10m32.296s
user 9m55.045s
sys 0m4.894s


:c00l:

Edit: and the trampoline doesn't perform well either
Haha, thats the same page that I found the snippet I posted on. Your version beats theirs on elegance as well.

SlightlyMadman
Jan 14, 2005

A A 2 3 5 8 K posted:

dictionary = ['word1','word2','word3']

if word.lower() in dictionary:
# try again

Haha, I think that list might be a little bit inefficient for storing every word in the dictionary, but thanks. I actually finally came across an enchant module which works very well.

edit: actually, my problem is still not 100% resolved. If a word is very close to a dictionary word, (i.e. "sh1t") it could still get through. It seems like this would be a common enough task that there should be a library specifically for it?

SlightlyMadman fucked around with this message at 18:31 on Oct 30, 2009

tef
May 30, 2004

-> some l-system crap ->

SlightlyMadman posted:

I have a specialized url shortener Django app that works on base64 translations of a database ID.

It might be easier to come up with an encoding that won't generate offensive language, than to repeatedly generate new ids if they happen to look like words.

SlightlyMadman
Jan 14, 2005

tef posted:

It might be easier to come up with an encoding that won't generate offensive language, than to repeatedly generate new ids if they happen to look like words.

Can you suggest any algorithms specifically? It's a url shortener, so it would need to at least be close to base64 in efficiency (both in number of characters and speed to encode/decode).

I've implemented a simple levenshtein algorithm in the meantime, but it only counts the number of letters a word is off by, not similarities in the letters themselves (i.e. "5h1t" is as far from "poo poo" as "xhbt" is).

Janitor Prime
Jan 22, 2004

PC LOAD LETTER

What da fuck does that mean

Fun Shoe
http://thedailywtf.com/Articles/The-Automated-Curse-Generator.aspx

Sorry I don't have anything else to add :(. I mean that guys solution could have been ok if he'd just used consonants.

tef
May 30, 2004

-> some l-system crap ->

tripwire posted:

Haha, thats the same page that I found the snippet I posted on. Your version beats theirs on elegance as well.

The trampoline version is still running :(

p.s your chaos maps are drat awesome

NadaTooma
Aug 24, 2004

The good thing is that everyone around you has more critical failures in combat, the bad thing is - so do you!

tef posted:

It might be easier to come up with an encoding that won't generate offensive language, than to repeatedly generate new ids if they happen to look like words.

Another option is Base-32:
http://en.wikipedia.org/wiki/Base32
http://www.crockford.com/wrmg/base32.html

A nice side-effect is that "u" isn't part of the encoding, so generating swear words becomes more difficult.

Edit: Looks like there's already support for this in Python!
http://docs.python.org/library/base64.html
Jjust search that page for "b32".

NadaTooma fucked around with this message at 20:23 on Oct 30, 2009

SlightlyMadman
Jan 14, 2005

NadaTooma posted:

Another option is Base-32:
http://en.wikipedia.org/wiki/Base32
http://www.crockford.com/wrmg/base32.html

A nice side-effect is that "u" isn't part of the encoding, so generating swear words becomes more difficult.

Edit: Looks like there's already support for this in Python!
http://docs.python.org/library/base64.html
Jjust search that page for "b32".

That's a good idea, but it would make my IDs twice as long. Since the actual point of url generation isn't as much as a bottleneck as resolution, I don't see a problem in using a more efficient encoding at the cost of a bit of analysis.

Habnabit
Dec 30, 2007

lift your skinny fists like
antennas in germany.
So I've long been sick of MySQLdb for being a horrible mysql driver as well as just being horribly written in general. Of course, there's also frequent questions on #python regarding either how horrible MySQLdb is or why the maintainer hasn't released any 2.6 binary packages for windows.

To solve all of this, I wrote my own mysql driver from scratch using Cython: oursql! It's still very much in-development and I haven't finished writing a comprehensive tutorial, but it works pretty well as far as I've tested. There's also a windows binary package for python 2.6!

It's a standard DB-API driver, so you can use it mostly the same way as MySQLdb previously. I haven't tested to see if you can easy_install it yet. Packages for the main package manager systems are being submitted as I figure out how to write them.

Hopefully there's some folks in this thread who are as tired of MySQLdb as I was and don't mind testing out a new driver.

nonathlon
Jul 9, 2004
And yet, somehow, now it's my fault ...

Habnabit posted:

To solve all of this, I wrote my own mysql driver from scratch using Cython: oursql! It's still very much in-development and I haven't finished writing a comprehensive tutorial, but it works pretty well as far as I've tested. There's also a windows binary package for python 2.6!

Good on you. I've had constant problems with installing MySQLdb, problems that shift and change: it won't install on OSX, it won't install on Fedora, it puts the libraries in the wrong place, configs wrong or (worse of all) installs and looks functional but isn't. So I'm relieved to see a possible replacement.

Any reason for using Cython, or does it just make things easier?

Habnabit
Dec 30, 2007

lift your skinny fists like
antennas in germany.

outlier posted:

Good on you. I've had constant problems with installing MySQLdb, problems that shift and change: it won't install on OSX, it won't install on Fedora, it puts the libraries in the wrong place, configs wrong or (worse of all) installs and looks functional but isn't. So I'm relieved to see a possible replacement.

Any reason for using Cython, or does it just make things easier?

While I'm definitely comfortable writing stuff in C, Cython removes all of the boilerplate associated with writing C extensions for python. It means I don't have to think about class structures, method structures, documentation structures, refcounts, exceptions, or the refcounts of everything I'm working with when an exception is raised. In the end, I write less code that has the potential to really screw up python.

So yes: I use Cython to make things easier.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Habnabit posted:

So I've long been sick of MySQLdb for being a horrible mysql driver as well as just being horribly written in general. Of course, there's also frequent questions on #python regarding either how horrible MySQLdb is or why the maintainer hasn't released any 2.6 binary packages for windows.

To solve all of this, I wrote my own mysql driver from scratch using Cython: oursql! It's still very much in-development and I haven't finished writing a comprehensive tutorial, but it works pretty well as far as I've tested. There's also a windows binary package for python 2.6!

It's a standard DB-API driver, so you can use it mostly the same way as MySQLdb previously. I haven't tested to see if you can easy_install it yet. Packages for the main package manager systems are being submitted as I figure out how to write them.

Hopefully there's some folks in this thread who are as tired of MySQLdb as I was and don't mind testing out a new driver.

Good for you! Awhile back I almost got to the point where I was going to teach myself how to do the same. Instead, I figured out how to compile MySQLdb for Windows myself (get it here if anyone needs it).

I'll be keeping an eye on your project, though.

the chip
Mar 10, 2002

Fun Shoe
Just got my very first python script running. It's just a Word of the Day, with a word list stored in a text file, and a "lookup" button that opens the definition in the web browser. Being the first thing I've done in python, I want to make sure my code is nice and pythonic, as it were. Any suggestions on how to clean it up?

code:
import wx
import random
import linecache
import webbrowser

class Frame(wx.Frame):
    numwords=0
    def __init__(self, parent, id):
        wx.Frame.__init__(self, parent, id, 'Word of the Day', size=(300,220))
        panel=wx.Panel(self)
        
        nextbutton=wx.Button(panel, label="Next", pos=(230,150), size=(50, 20))
        lookupbutton=wx.Button(panel, label="Lookup", pos=(20,150), size=(50, 20))
        self.Bind(wx.EVT_BUTTON, self.nextWord, nextbutton)
        self.Bind(wx.EVT_BUTTON, self.lookup, lookupbutton)
        self.Bind(wx.EVT_CLOSE, self.closeWindow)
        
        self.text1 = wx.TextCtrl(panel, -1, "Click Next", pos=(20, 20))
        
    def closeWindow(self, event):
        self.Destroy()
        
    def lookup(self, event):
        webbrowser.open_new('http://dictionary.reference.com/browse/'+self.text1.GetValue())
        
    def nextWord(self, event):
        wordchoice=linecache.getline("D:/dictionary.txt", random.randint(0, self.numwords))
        self.text1.SetValue(str(wordchoice))
        
    def setNumWords(self, numwords):
        self.numwords=numwords
        
if __name__=='__main__':
    i=open("D:/dictionary.txt","r")
    numwords=len(i.readlines())
    
    app=wx.PySimpleApp()
    frame=Frame(parent=None, id=-1)
    frame.setNumWords(numwords)
    frame.Show()
    app.MainLoop()

BeefofAges
Jun 5, 2004

Cry 'Havoc!', and let slip the cows of war.

Depending on how bored you are, read the Python style guide. http://www.python.org/dev/peps/pep-0008/

tripwire
Nov 19, 2004

        ghost flow
Well it looks fine to me, but why use the linecache module? The biggest wordlists I can see on packetstorm are like a megabyte, so unless memory is at an extreme premium, why not load the word list from the file once?

I don't think theres a lot of overlap with most user interface libraries like tkinter or wxwidgets and "pythonic" code; no matter how you slice it, you'll always get a lot of verbose boilerplate in there. Some may disagree.

Also, the random module provides a handy "choice" function that returns a random element from any sequence type, just for cases like this.
If I want a random word I'll usually do something like this:

code:
from random import choice

with open('dictionary.txt','r') as my_file:
    file_contents = my_file.read()
    word_list     = tuple(word.strip() for word in file_contents.split('\n') )
    

def get_new_word():
    return choice(word_list)

You could use a list comprehension rather than the generator I've used above, it doesn't make a big difference in this case, but in general I try to avoid using mutable sequence types if the data contained isn't going to change.

the chip
Mar 10, 2002

Fun Shoe
Thanks tripwire. I learned alot from that.

Avenging Dentist
Oct 1, 2005

oh my god is that a circular saw that does not go in my mouth aaaaagh
Ok design question: I have several related C interfaces which may or may not be present in a given implementation. The interfaces look like:
code:
void Foo_a(Foo_instance f,...);
void FooBar_b(Foo_instance f,...);
void FooExt_c(Foo_instance f,...);
...
Which are then turned into a Python class somehow. This is easy when all I have is the Foo interface, or Foo+FooBar (make FooBar a subclass), but not so much when I want to make arbitrary combinations of the interfaces. I can see two options (though maybe there's a better one than either of these):

A) For FooBar/FooExt, create modules that inject new methods into the base Foo class. e.g.
code:
import Foo # gives us Foo.Foo and Foo.Foo.a
import FooBar # gives us Foo.Foo.b
B) Create FooBar and FooExt as subclasses of Foo and just force the user to deal with the poo poo. This has the side-effect of likely imposing another pointer-dereference for every function (for reasons that aren't worth getting into), but then I probably shouldn't be worrying about performance at the Python-binding level anyway.

Habnabit
Dec 30, 2007

lift your skinny fists like
antennas in germany.

Avenging Dentist posted:

Ok design question: I have several related C interfaces which may or may not be present in a given implementation. The interfaces look like:
code:
void Foo_a(Foo_instance f,...);
void FooBar_b(Foo_instance f,...);
void FooExt_c(Foo_instance f,...);
...
Which are then turned into a Python class somehow. This is easy when all I have is the Foo interface, or Foo+FooBar (make FooBar a subclass), but not so much when I want to make arbitrary combinations of the interfaces. I can see two options (though maybe there's a better one than either of these):

A) For FooBar/FooExt, create modules that inject new methods into the base Foo class. e.g.
code:
import Foo # gives us Foo.Foo and Foo.Foo.a
import FooBar # gives us Foo.Foo.b
B) Create FooBar and FooExt as subclasses of Foo and just force the user to deal with the poo poo. This has the side-effect of likely imposing another pointer-dereference for every function (for reasons that aren't worth getting into), but then I probably shouldn't be worrying about performance at the Python-binding level anyway.

Really I'm not seeing why you can't just put it all in one class. What's the overlap between Foo_a, FooBar_b, and FooExt_c?

Avenging Dentist
Oct 1, 2005

oh my god is that a circular saw that does not go in my mouth aaaaagh

Habnabit posted:

Really I'm not seeing why you can't just put it all in one class. What's the overlap between Foo_a, FooBar_b, and FooExt_c?

If I put it in one class (without injection) I'd be relying entirely on conditional compilation to find and include the appropriate C interfaces which would a) be pretty loving confusing for users trying to write stuff that works across implementations (you wouldn't want to use FooMyImplementationsSpecialStuff_d), and b) would break the ability for other people to add new interfaces modifying the base interface (Foo) without recompiling the entire package.

Habnabit
Dec 30, 2007

lift your skinny fists like
antennas in germany.

Avenging Dentist posted:

If I put it in one class (without injection) I'd be relying entirely on conditional compilation to find and include the appropriate C interfaces which would a) be pretty loving confusing for users trying to write stuff that works across implementations (you wouldn't want to use FooMyImplementationsSpecialStuff_d), and b) would break the ability for other people to add new interfaces modifying the base interface (Foo) without recompiling the entire package.

There's a precedent for just using conditional compilation that's even in the python stdlib; look at things like the fcntl or socket modules.

Anyway, the solution that makes the most sense to me would be to have subclasses to represent the different extensions, assuming that you wouldn't use more than one extension at a time. Otherwise I'd just jam it all in one class and provide some sort of compatibility layer on top.

sonic bed head
Dec 18, 2003

this is naturual, baby!
I just downloaded pydev for eclipse and I am having a lot of trouble figuring out why I get unresolved imports when the code runs without errors. The code that I'm working with is very basic and the problem still happens.

code:
from xml.dom.minidom import parseString
import httplib;
print httplib;
That runs with no errors from pydev, but parseString and httplib are underlined with red and show as errors in the error console of eclipse. I have one interpreter setup in the eclipse preferences, which is just the osx default interpreter at /usr/bin/python. I have the python path set to the python framework folder, /System/Library/Frameworks/Python/Versions/2.5/lib/Python2.5. I have tried adding httplib and minidom as builtins, and the problem is still there. Any idea what I can do? Thanks for your help.

m0nk3yz
Mar 13, 2002

Behold the power of cheese!
FYI: The talks for PyCon have been announced: http://us.pycon.org/2010/conference/talks/

Ya'll should come.

king_kilr
May 25, 2007

m0nk3yz posted:

FYI: The talks for PyCon have been announced: http://us.pycon.org/2010/conference/talks/

Ya'll should come.

Yeah you should, it's going to kick rear end.

Lurchington
Jan 2, 2003

Forums Dragoon
I want to see talks: 13, 30, 48, *52*, *55*, 64, 81, 93, 96, 116, 171, 186, 188

  • 13 - How and why Python is being used to by the Military to model real-world battlefield scenarios
  • 30 - Python Testing Patterns
  • 48 - Python 3: The Next Generation
  • 52 - New *and* Improved: Coming changes to unittest, the standard library test framework
  • 55 - The Mighty Dictionary
  • 64 - Python Metaprogramming
  • 81 - rapid multi-purpose testing
  • 93 - PLY and PyParsing
  • 96 - Introduction to unittest (a.k.a. PyUnit)
  • 116 - Cross platform application development and distributio
  • 171 - Customizing your editor for maximum productivity
  • 186 - Powerful Pythonic Patterns
  • 188 - Tests and Testability

I think my company may be interested in sending me :)

Lurchington fucked around with this message at 18:45 on Nov 5, 2009

ATLbeer
Sep 26, 2004
Über nerd

m0nk3yz posted:

FYI: The talks for PyCon have been announced: http://us.pycon.org/2010/conference/talks/

Ya'll should come.

Considering I can walk to the conference.. I should be there

Already on my to see list
  • Distributed Programming with Pyro
  • Cooperative Multitasking with Twisted: Getting Things Done Concurrently.
  • Python Testing Patterns
  • Simulating network devices with Python
  • Managing the world's oldest Django project
  • Why not run all your tests all the time? A study of continuous integration systems
  • Powering the real-time web with Python: PubSubHubbub on App Engine

Adbot
ADBOT LOVES YOU

nbv4
Aug 21, 2002

by Duchess Gummybuns
virtualenv nub here. Whats the recommend way to install a module from svn or git or something? Do you just navigate to the /site-packages/ directory within the virtualenv then do a checkout?

  • Locked thread