Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
m0nk3yz
Mar 13, 2002

Behold the power of cheese!

Lurchington posted:

I've been writing stuff in python pretty much every day for a year and it's only now coming to my attention how little of the language I'm actually using day-to-day. Context management seems exciting and I'm looking forward to getting better.

I think this was floated earlier in reference to podcasts, but what are the python RSS I should consider staying with. I think I'm interested in items similar to "here's a feature you might be under-utilizing, and here's why it's useful."

I've been writing python pretty much full time for about 6-7 years now, and I still don't do it as well, or take advantage of, some of the cooler features/language bits that I see others use. That's one of the big reason why I tend to read through the implementation of core modules in python SVN - they teach you a lot.

Adbot
ADBOT LOVES YOU

nihilocrat
Jun 8, 2004

all the hep cats jam to "cat /dev/hda1 > /dev/audio"
I'd say learning the internals and standard conventions is pretty helpful, non-obvious stuff which is useful but you might not naturally learn or use. You can do pretty cool stuff fiddling around with anything that comes between two sets of "__".

The 'conventions' that I randomly picked up by reading other peoples' code makes me realize why Python OOP is more "pure" than C++ or Java, but why a C++ or Java programmer will think Python OOP is horribly broken and tacked-on.

edit:

bitprophet posted:

See, I just DO NOT GET why everyone has this huge stiffie for developing (sans runserver and the builtin Python, of course -- which is fine for a newbie) on a local OS X workstation. It's loving retarded -- you have to wrestle with 3 or 4 different potential methods of installing the various components and then you end up with 2 versions of just about everything (the Apple default and the new one you installed because the Apple default is too loving old or lovely).

I do all my dev on standalone Linux servers, or VMWare Fusion running the same. aptitude install libapache2-mod-python python-psycopg2 postgresql-server and those (and their automatically installed dependencies) get me 95% of the way there, no fuss, no muss. An svn checkout of Django and I'm pretty much all done.

Just kills me that people put so much effort into Mac-native dev; my office is largely a Rails shop and they do the same thing, drives me bonkers.

/derail

I have the same feelings about RHEL / CentOS. To me, "enterprise" seems to mean "really loving old packages which you aren't going to use, and instead compile your own versions of anything important and completely defeat the purpose of using a packaging system".

Macs in general just confuse me. Why spend a lot on some alien artifact-looking machine when you can just scrounge up some old hardware, slap Linux on it, and run with it? Seriously, if you are doing web development you aren't really going to need any significant power on your workstation.

duck monster posted:

Oh god.... One of the angriest emails I ever wrote was something to the effect of "Why the gently caress are you compiling free software with an unfree compiler? USE GCC!!!!!!!!" because I was blocked from using so many libraries due to the need for Visual studio.

Hey, that's one of the main reasons I avoid C/C++ these days. People think I'm a pussy, but at least I can get "Hello World" without having to find/compile/link to a zillion dependency libraries.

nihilocrat fucked around with this message at 17:58 on Oct 6, 2008

UberJumper
May 20, 2007
woop
Using ftplib is there anyway to resume binary file uploads?

I've looked and looked, and i can't find anything. REST doesn't seem to work, when uploading to the server. Storing binary on the server, always causes it to start writing from the beginning of the file again.

nonathlon
Jul 9, 2004
And yet, somehow, now it's my fault ...

nihilocrat posted:

Hey, that's one of the main reasons I avoid C/C++ these days. People think I'm a pussy, but at least I can get "Hello World" without having to find/compile/link to a zillion dependency libraries.

Actually I agree. I programmed in C++ for years and recently had to return to it. The whole library-link thing is ridiculous: compilers being unable to find a library you have on your disk, or compiling and then complaining about an "unresolved symbol". If nothing else, Java got all that stuff right.

m0nk3yz posted:

.. on context managers ...

Well, huh. Thanks for the explanation.That's actually useful - I actually constructs like that in C++ (set something up in the c'tor, clean it up in the d'tor) and recently tried to do something similar in Python. (It works but only 2 problems - you can't tell when the object will be deleted, and session termination disposes of objects in a weird way). This will be a lot easier.

duck monster
Dec 15, 2004

My head hurts.

What the hell were apple thinking even CONTEMPLATING distributing python without loving database bindings. Without the db bindings, almost the entire spectrum of useful things to do with scripting languages is left a flibbering mess.

Lazy Apple. Lazy loving lazy.

Anyone know a good way of getting mysqldb of some variety working on the mac. This is making me too angry working it out for myself :(

bitprophet
Jul 22, 2004
Taco Defender
Macports' version not working? (I forget if we went over this already in the Django thread :v:)

m0nk3yz
Mar 13, 2002

Behold the power of cheese!

duck monster posted:

My head hurts.

What the hell were apple thinking even CONTEMPLATING distributing python without loving database bindings. Without the db bindings, almost the entire spectrum of useful things to do with scripting languages is left a flibbering mess.

Lazy Apple. Lazy loving lazy.

Anyone know a good way of getting mysqldb of some variety working on the mac. This is making me too angry working it out for myself :(

Apple doesn't distribute mysql - why would they give you bindings, especially given those bindings compile against a particular version of MySQL that's local to the machine?

Note, I do have instructions, I just need to find them - I'll post them when I have them

Edit: Here you go: http://yousefourabi.com/apple/django-on-leopard

Note, that when he says edit _mysql.c - you need to change 2 parts, which he omits. Your diff should look like this:
code:
$ diff _mysql.c ../_mysql.c 
37,39d36
< #ifndef uint
< #define uint unsigned int
< #endif
484,485c481,482
< 	uint port = MYSQL_PORT;
< 	uint client_flag = 0;
---
>     unsigned int port = MYSQL_PORT;
>     unsigned int client_flag = 0;
Let me know if you need more - I can compile, import it and do all sorts of cool stuff with it. You need to install the 32 bit version of mysql though.

m0nk3yz fucked around with this message at 22:15 on Oct 7, 2008

nonathlon
Jul 9, 2004
And yet, somehow, now it's my fault ...

duck monster posted:

Anyone know a good way of getting mysqldb of some variety working on the mac. This is making me too angry working it out for myself :(

Getting MySQLdb to work can be a bitch. Sometimes it just works but other times I've spent days trying to get it installed (with occasional periods of it looking like it installed but it actually not working).

General advice: use of the precompiled packages out there. Or use Postgres instead. Or follow the tale of woe on my webpages as I encounter every problem under the sun.

I've also had problems with MySQLdb on Redhat as well, so it's not just Mac. DPAPI is okay, but by now Python really needs a saner and more standard set of db modules, like Java.

duck monster
Dec 15, 2004

m0nk3yz posted:

Apple doesn't distribute mysql - why would they give you bindings, especially given those bindings compile against a particular version of MySQL that's local to the machine?

Note, I do have instructions, I just need to find them - I'll post them when I have them



I'd argue however that Mysql is fundamental enough that apple ought be providing its own package of that as well, or at least providing bindings against the mysql downloadable from the mysql site.

Because could you imagine distributing a python app for , say cocoa, that had to do mysql or postgres work?

At least with windows, you have mysql.dll or whatever it is, and it all seems to magically kinda work.

m0nk3yz
Mar 13, 2002

Behold the power of cheese!

duck monster posted:

I'd argue however that Mysql is fundamental enough that apple ought be providing its own package of that as well, or at least providing bindings against the mysql downloadable from the mysql site.

Because could you imagine distributing a python app for , say cocoa, that had to do mysql or postgres work?

At least with windows, you have mysql.dll or whatever it is, and it all seems to magically kinda work.

Did the instructions I provided help you?

Habnabit
Dec 30, 2007

lift your skinny fists like
antennas in germany.

duck monster posted:

I'd argue however that Mysql is fundamental enough that apple ought be providing its own package of that as well, or at least providing bindings against the mysql downloadable from the mysql site.

Because could you imagine distributing a python app for , say cocoa, that had to do mysql or postgres work?

At least with windows, you have mysql.dll or whatever it is, and it all seems to magically kinda work.

Ever worked with py2app? It would take a little more work than on Windows, sure, but if you use py2app to make the distribution and then use otool to change the _mysql.so file to use a package-local version of libmysqlclient_r.dylib instead of one in /usr/local/lib or whatever, then you'd be fine.

I haven't done this myself, but there's no reason it shouldn't work as long as you include both _mysql.so and the client dylib.

e: Assuming, of course, that it really is that simple on Windows.

Habnabit fucked around with this message at 01:16 on Oct 10, 2008

latexenthusiast
Apr 22, 2008
Total "RTFM, n00b!!" type question, but I read the manual, and I still have no clue. In fact, you may go so far as to suggest that if I'm asking such a question, I have no business writing code that needs to be documented anyway, but whatever.

So I installed Epydoc, as per the manual's suggestion:
code:
matlocks-powerbook:~ matlock$ cd ./epydoc-3.0.1/
matlocks-powerbook:~/epydoc-3.0.1 matlock$ su
Password:
matlocks-powerbook:/Users/matlock/epydoc-3.0.1 root# python setup.py install
running install
running build
running build_py
running build_scripts
running install_lib
running install_scripts
changing mode of /Library/Frameworks/Python.framework/Versions/2.5/bin/.DS_Store to 755
changing mode of /Library/Frameworks/Python.framework/Versions/2.5/bin/apirst2html.py to 755
changing mode of /Library/Frameworks/Python.framework/Versions/2.5/bin/epydoc to 755
changing mode of /Library/Frameworks/Python.framework/Versions/2.5/bin/epydocgui to 755
running install_egg_info
Removing /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/epydoc-3.0.1-py2.5.egg-info
Writing /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/epydoc-3.0.1-py2.5.egg-info
although in the manual, the 755 seems to be a 100755 instead.

So my problem is this: how do I launch it from the command line? I'm relatively new to using the bash, but I tried looking up how to launch scripts in Unix, and that did nothing for me. I just want to document my code for a class on applied numerical methods!

latexenthusiast fucked around with this message at 08:23 on Oct 11, 2008

tef
May 30, 2004

-> some l-system crap ->
The command should be " /Library/Frameworks/Python.framework/Versions/2.5/bin/epydoc"

m0nk3yz
Mar 13, 2002

Behold the power of cheese!
Also, do the following - open your ~/.bash_profile or ~/.bashrc in your favorite editor - this file controls your bash environment. For me, my ~/.bashrc file contains:

code:
. ~/.bash_profile
Because I can never remember which one OS/X uses.

Add the following line to your .bash_profile:
code:
export PATH=$PATH:/Library/Frameworks/Python.framework/Versions/2.5/bin/
And save the file. When you open a new terminal window or type ". ~/.bash_profile" - that new bin directory will be added to your PATH, and all you will need to do is type "epydoc" on the command line.

Epydoc is awesome - I just cut work over to using it for our python tests and code. I just can't get the UML charts to render for some odd reason.

latexenthusiast
Apr 22, 2008
You guys are the best. Thank you so much. I thought there was no way I'd have to type in the full path, given how long it was, but I guess I was wrong. Dead wrong. The tip about .bash_login was really helpful, too, so thanks!

m0nk3yz
Mar 13, 2002

Behold the power of cheese!
FWIW, I chucked another side project up on pypi today:

http://pypi.python.org/pypi/pyjavaproperties/

Here's the others:
http://pypi.python.org/pypi/nose-testconfig/
http://code.google.com/p/testbutler/

I've got two more pending nose plugin projects as well as a multiprocessing/MPI package coming as well, although the last one might turn into an actor-model thing, right now I am trying to pull dramatis (http://dramatis.mischance.net/) apart.

duck monster
Dec 15, 2004

m0nk3yz posted:

FWIW, I chucked another side project up on pypi today:

http://pypi.python.org/pypi/pyjavaproperties/

Here's the others:
http://pypi.python.org/pypi/nose-testconfig/
http://code.google.com/p/testbutler/

I've got two more pending nose plugin projects as well as a multiprocessing/MPI package coming as well, although the last one might turn into an actor-model thing, right now I am trying to pull dramatis (http://dramatis.mischance.net/) apart.

woo testbutler looks nice.

m0nk3yz
Mar 13, 2002

Behold the power of cheese!

duck monster posted:

woo testbutler looks nice.

It's still in the prototype stages, although we have it deployed for actual production use at work. I have a laundry list of things I want to add to it. Right now I am working on a plugin for nose to upload results to the system in a new testbutler.results app.

Tomed2000
Jun 24, 2002

Is there an efficient way to remove all non-alphanumeric characters from strings without using regular expressions?

tef
May 30, 2004

-> some l-system crap ->
Yes, but:

Why efficient - strings are immutable in python, so often any character by character option won't be that efficient

Why not regular expressions - they're in the standard library ?

hey mom its 420
May 12, 2007

One way to do it:
code:
>>> reduce(lambda x,y:x+y, [x for x in "Hey, there!" if x.isalnum()])
'Heythere'
But yeah, use regexen.
code:
>>> import re
>>> re.sub(r'[^a-zA-Z0-9]','','aaa!haha!!!,,,')
'aaahaha'

hey mom its 420 fucked around with this message at 00:32 on Oct 13, 2008

Pivo
Aug 20, 2004


Tomed2000 posted:

Is there an efficient way to remove all non-alphanumeric characters from strings without using regular expressions?

code:
>>> import string
>>> s = "1@2#3$4%5^6&7*8(9)0-=abcdefghijklmnopetcABCDEFGHIJKLKMNOyougetthepoint"
>>> filter(lambda c: c in string.letters or c in string.digits, s)
'1234567890abcdefghijklmnopetcABCDEFGHIJKLKMNOyougetthepoint'
Seems like it would do a lot of comparisons though.


edit: \/\/ this one is better

Pivo fucked around with this message at 00:35 on Oct 13, 2008

Scaevolus
Apr 16, 2007

code:
>>> filter(str.isalnum, "Hey, there")
'Heythere'

Allie
Jan 17, 2004

That's definitely a contrived way of using reduce(), and I can't imagine it being efficient. I'd use str.join:
code:
>>> ''.join([c for c in 'Hey, there!' if c.isalnum()])
'Heythere'
But I think a regular expression would work best.

hey mom its 420
May 12, 2007

Ugh, I've been in Haskell land for too long, that reduce was uncalled for.

Scaevolus
Apr 16, 2007

Pivo posted:

code:
>>> import string
>>> s = "1@2#3$4%5^6&7*8(9)0-=abcdefghijklmnopetcABCDEFGHIJKLKMNOyougetthepoint"
>>> filter(lambda c: c in string.letters or c in string.digits, s)
'1234567890abcdefghijklmnopetcABCDEFGHIJKLKMNOyougetthepoint'

This has O(n2) complexity.

Milde posted:

code:
>>> ''.join([c for c in 'Hey, there!' if c.isalnum()])
'Heythere'
There's no need to make a temporary list.
code:
>>> ''.join(c for c in 'Hey, there!' if c.isalnum())
'Heythere'

Pivo
Aug 20, 2004


Scaevolus posted:

This has O(n2) complexity.

How come? Is it doing string concatenation at every step of the filter? I'm a Python neophyte..

Scaevolus
Apr 16, 2007

Pivo posted:

How come? Is it doing string concatenation at every step of the filter? I'm a Python neophyte..
character in string is O(n).

Allie
Jan 17, 2004

Scaevolus posted:

This has O(n2) complexity.

There's no need to make a temporary list.
code:
>>> ''.join(c for c in 'Hey, there!' if c.isalnum())
'Heythere'

Technically the generator expression would be slower, but use less memory, or something like that. The difference is miniscule for that particular string, and it looks like the filter() method is faster than both.

The regular expression would probably be the fastest though, and it's a perfect fit.

Pivo
Aug 20, 2004


Scaevolus posted:

character in string is O(n).

string.digits and string.letters are constant length though assuming locale doesn't change mid-execution.

Our n is the len(s) where s is the input string. So, should still be linear.

Scaevolus
Apr 16, 2007

Pivo posted:

string.digits and string.letters are constant length though assuming locale doesn't change mid-execution.

Our n is the len(s) where s is the input string. So, should still be linear.
Oops, O(nm) then ;)

Lurchington
Jan 2, 2003

Forums Dragoon
On the subject of using Regular expressions to remove characters. Is there a good method of removing (or replacing) any characters considered illegal-filename characters?

I know those characters are different between Windows and Unix, and so far I manually specified those characters, but I'd prefer something else if I can help it. Some google searches returned that it's impossible to know illegal characters ahead of time, which is surprising to me.

tef
May 30, 2004

-> some l-system crap ->
the problem is that you often don't know the filesystem a directory is using, and so even under unix you can be subject to windows filenames.

what's wrong with try/catch instead of trying to guess what might be valid?

Lurchington
Jan 2, 2003

Forums Dragoon

tef posted:

the problem is that you often don't know the filesystem a directory is using, and so even under unix you can be subject to windows filenames.

what's wrong with try/catch instead of trying to guess what might be valid?

are there different exceptions for a name issue vs. permissions vs. out of space? If so, I can try it. It just seemed clunky and wanted to know what other options I had.

Allie
Jan 17, 2004

Take a look at OSError and error values in errno. You can compare the exception's errno attribute to the values in the errno library. Also, if you try to call open() with a path containing a null byte, it'll raise a TypeError, at least on OS X.

tef is right though, this isn't something you can know with great certainty in advance. For example, HFS+ on OS X will percent-encode invalid characters in filenames. ext3 will allow pretty much anything, even paths invalid for the file system's encoding. And sys.getfilesystemencoding() isn't always accurate when you have multiple file systems.

I think what you do depends on your application. If you're presenting a save file dialog, you should try the operation, and if it fails, present an error to the user. If you're trying to save arbitrary filenames for something like a database of files, it might be better to encode the filenames beforehand.

MononcQc
May 29, 2007

I'm using MySQLdb to fetch many rows from a table, and all of them contain many fields including DATETIME columns.

code:
(
    {'body_text': 'some text here', 
     'headline': 'hello title', 
     'author': 'Ferd', 
     'id': 13L, 
     'published': 1, 
     'pub_date': datetime.datetime(2008, 9, 25, 22, 6, 26),
     'slug': 'hello-title'
    },
    {'body_text': 'some text here again', 
     'headline': 'hello second example', 
     'author': 'Ferd', 
     'id': 12L, 
     'published': 1, 
     'pub_date': datetime.datetime(2008, 9, 25, 22, 6, 26),
     'slug': 'hello-second-title'
    },
)
I'm then taking this result and trying to use it inside a json function. The problem here is the datetime.datetime object.

Both cjson and python-json throw an error at it. What I'd need is the string version of it so it can be jsonised properly.

I'm looking for a way to either have MySQLdb automatically format DATETIME columns when returning them, or to efficiently go through the tuple of dicts and call .__str__() or str() on the objects. Or have a json writer/encoder do it for me.

Is this doable or am I going to need to change the way I store poo poo to use unix timestamps instead?

TOO SCSI FOR MY CAT
Oct 12, 2008

this is what happens when you take UI design away from engineers and give it to a bunch of hipster art student "designers"
There's no standard format for date/times in JSON. You'll need to decide what format to use (ISO 8601 is a good choice) and then override serialization in your JSON encoder for that type.

Here's an encoder for simplejson:

code:
class DateJSONEncoder(simplejson.JSONEncoder):
    def default(self, o):
        if isinstance(o, datetime.datetime):
            return o.strftime("%Y-%m-%dT%H:%M:%S")
        return super(DateJSONEncoder, self).default(o)

simplejson.dumps(data, cls = DateJSONEncoder)
Similar code would be written for demjson or jsonlib. Don't use python-json or cjson, they're loving terrible.

TOO SCSI FOR MY CAT fucked around with this message at 05:58 on Oct 14, 2008

Lurchington
Jan 2, 2003

Forums Dragoon

Milde posted:

I think what you do depends on your application. If you're presenting a save file dialog, you should try the operation, and if it fails, present an error to the user. If you're trying to save arbitrary filenames for something like a database of files, it might be better to encode the filenames beforehand.

a re-write of my tv-episode bulk renaming app* that will rename files using a user-defined template and information (eg, show name, show title) that has a decent chance of containing illegal characters. And I haven't even accounted for unicode (specifically foreign characters) yet.

My original plan was to encode the filenames beforehand, probably with the most common characters removed, and avoid any human interaction as much as possible. It may not be smart to avoid it in this case. I just want need to come up with a smart way of not forcing too much interaction. For example I really don't want the user to manually correct 60 instances of Avatar: The Last Airbender.

Thanks for responses, and if there are any suggestions, I'd love to hear it.

* When I'm happy that it's not devoid of anything resembling best-practices I'd love to do a code-review type thing. This has been my learn-python project for over a 6 months now.

Bozart
Oct 28, 2006

Give me the finger.

Lurchington posted:

a re-write of my tv-episode bulk renaming app* that will rename files using a user-defined template and information (eg, show name, show title) that has a decent chance of containing illegal characters. And I haven't even accounted for unicode (specifically foreign characters) yet.

My original plan was to encode the filenames beforehand, probably with the most common characters removed, and avoid any human interaction as much as possible. It may not be smart to avoid it in this case. I just want need to come up with a smart way of not forcing too much interaction. For example I really don't want the user to manually correct 60 instances of Avatar: The Last Airbender.

Thanks for responses, and if there are any suggestions, I'd love to hear it.

* When I'm happy that it's not devoid of anything resembling best-practices I'd love to do a code-review type thing. This has been my learn-python project for over a 6 months now.

A dead simple way is to just replace all illegal characters with _'s with a regex. Who cares about encoding if the result is going to be

Avatarx80 the last airbender

or whatever. Avatar_the_last_airbender is going to work pretty much anywhere.

Adbot
ADBOT LOVES YOU

Lurchington
Jan 2, 2003

Forums Dragoon
Which I'm fine with doing as long as I know which characters are illegal, which I thought was the issue. I need to investigate OSError, but will that report which characters are illegal?

Is there a list of characters that are illegal in at least one filesystem? I'm fine with losing some possibly legit characters in the crossfire for at least the first go.

  • Locked thread