Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

ohgodwhat: Aug 6, 2005

Yup. I don't think they take it tremendously seriously as anything but a way to weed out idiots who come out if CS without knowing what a for loop is.

EDIT: Wow, it's on Python 2.4

ohgodwhat fucked around with this message at 01:40 on Mar 7, 2014

# ? Mar 7, 2014 00:45

Adbot: ADBOT LOVES YOU

# ? Jun 2, 2024 17:28

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

On every deploy I blow away the old virtualenv and create a new one, and then pip install -r requirements.txt. This takes several minutes, how can I speed this up?

# ? Mar 7, 2014 03:56

BigRedDot: Mar 6, 2008

fletcher posted:

On every deploy I blow away the old virtualenv and create a new one, and then pip install -r requirements.txt. This takes several minutes, how can I speed this up?

Try conda instead of virtualenv/pip.

# ? Mar 7, 2014 04:00

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Alternatively, if download speeds or latency are the bottleneck, try hosting a local cache of the packages you usually install.

# ? Mar 7, 2014 04:20

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

BigRedDot posted:

Try conda instead of virtualenv/pip.

Sounds interesting. Not a lot of stars on github though, is it production ready?

BeefofAges posted:

Alternatively, if download speeds or latency are the bottleneck, try hosting a local cache of the packages you usually install.

Download speed seems great, I think it's compiling things like pycyrpto and uWSGI that's the bottleneck

# ? Mar 7, 2014 04:27

Lurchington: Jan 2, 2003; Forums Dragoon

fletcher posted:

On every deploy I blow away the old virtualenv and create a new one, and then pip install -r requirements.txt. This takes several minutes, how can I speed this up?

if you mean deploying using some sort of distributable (rpm or deb) it's not uncommon to bundle the virtualenv into the distributable. This means you can eat the time it takes to get everything in before you bring your site down to upgrade. This can be combined with the local cache BeefOfAges suggests.

If you mean you have some git checkout that you pull from after someone tagged, and that checkout is being run in production, it's clearly pretty manual. In that case, you may just avoid blowing away the virtualenv if you didn't add any dependencies, and you can get by with removing all the .pyc / __pycache__ files.

Hopefully you're not actually doing the second one if it's for a job, but I've done this for personal stuff before.

fletcher posted:

Sounds interesting. Not a lot of stars on github though, is it production ready?

Download speed seems great, I think it's compiling things like pycyrpto and uWSGI that's the bottleneck

a lot of the stuff that needs to be compiled is packaged pre-built on your distro's package manager (apt-get/yum). It will install to your system python, but you could set your virtualenv to allow site-packages if you really need it.

Lurchington fucked around with this message at 04:34 on Mar 7, 2014

# ? Mar 7, 2014 04:31

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

My requirements.txt is just a bunch of URLs to S3 where I've mirrored all the .tar.gz files from PyPI that I need. The deploy happens as part of a chef run, so I don't want to have to think about which packages I've added or removed vs. what is in the existing virtualenv, which is why I went down the route of deleting the virtualenv and recreating it every time. I'm hesitant to use the distro specific packages for my dependencies because it seems easier (for now) to control which version is installed by using pip & pypi packages.

# ? Mar 7, 2014 05:02

Plorkyeran: Mar 22, 2007; To Escape The Shackles Of The Old Forums, We Must Reject The Tribal Negativity He Endorsed

Build wheels for all of your packages?

# ? Mar 7, 2014 05:25

SYSV Fanfic: Sep 9, 2003; by Pragmatica

My program is corrupting files it copies, and I can't figure out why. I am using the OS module to compare time stamps and copy the files. I do open the files, but only for reading.

Code here

Its just a stupid script I wrote to be a ghetto steam cloud using google drive. I know blindly loading random files into memory is bad if those files are too large, but this is just a prototype.

# ? Mar 7, 2014 15:56

Pythagoras a trois: Feb 19, 2004; I have a lot of points to make and I will make them later.

I want to basically brute force all letter combinations of length n, in order. I made a generator that works pretty easily for each letter length:

code:

from string import ascii_lowercase

def one_letter():
    for i in ascii_lowercase:
        yield i

def two_letters():
    for i in ascii_lowercase:
        for j in ascii_lowercase:
            yield i + j

But how do I make a generator of variable letter length?

code:

def any_letters(max_length=5):
    output = ""
    while max_length >= len(output):
        yield ~magic~

I've tried describing the rules of iterating the letter combination up by hand, which turned out to be a huge pain in the rear end, and also counting up to sys.max_int and converting the decimal number into its letter combination, and the conversion logic got tangled pretty quickly.

Is there an easy way to do this? It seems like the thing 'for's and 'while's were designed to do.

Edit: I'm was trying to brute force keys for this guy, and realized I didn't know how to do this incredibly simple task.

Pythagoras a trois fucked around with this message at 16:43 on Mar 7, 2014

# ? Mar 7, 2014 16:39

KICK BAMA KICK: Mar 2, 2009

If you just want the result rather than "show your work" check out the permutation/combination functions in the itertools module.

# ? Mar 7, 2014 16:41

Nippashish: Nov 2, 2005; Let me see you dance!

Cheekio posted:

Is there an easy way to do this? It seems like the thing 'for's and 'while's were designed to do.

Something like this:

code:

def n_letters(n):
    if n == 0:
        return ""

    for i in ascii_lowercase:
        for tail in n_letters(n-1):
            yield i + tail

but in reality you should do what KICK BAMA KICK said and use itertools.

# ? Mar 7, 2014 16:46

Pythagoras a trois: Feb 19, 2004; I have a lot of points to make and I will make them later.

I wanted the work shown, but being pointed to the itertools module was actually the answer I needed. Thanks, I feel less stupid now.

Edit: I posted before I saw that recursive response. Thanks- I need to pull that apart to see why it works, I still don't quite get how to write recursive code.

Pythagoras a trois fucked around with this message at 17:31 on Mar 7, 2014

# ? Mar 7, 2014 16:47

evilentity: Jun 25, 2010

keyvin posted:

My program is corrupting files it copies, and I can't figure out why. I am using the OS module to compare time stamps and copy the files. I do open the files, but only for reading.

Code here

Its just a stupid script I wrote to be a ghetto steam cloud using google drive. I know blindly loading random files into memory is bad if those files are too large, but this is just a prototype.

I did something similar, basically mirror stuff to and from a pendrive.
This should do the trick:

Python code:

distutils.dir_util.copy_tree(src, dst, update=1)

Your mp5 is maybe wrong.

Python code:

def md5_of_file(path, block_size=2**20):
    md5 = hashlib.md5()
    with open(path, 'rb') as f: 
        for chunk in iter(lambda: f.read(128*md5.block_size), ''): 
             md5.update(chunk)
    return md5.digest()

# ? Mar 7, 2014 17:39

SYSV Fanfic: Sep 9, 2003; by Pragmatica

evilentity posted:

I did something similar, basically mirror stuff to and from a pendrive.
This should do the trick:
Python code:
distutils.dir_util.copy_tree(src, dst, update=1)  
Your mp5 is maybe wrong.
Python code:
def md5_of_file(path, block_size=2**20):
    md5 = hashlib.md5()
    with open(path, 'rb') as f: 
        for chunk in iter(lambda: f.read(128*md5.block_size), ''): 
             md5.update(chunk)
    return md5.digest()

I took the md5 operation completely out. It just relies on the file metadata now. Still not sure what is going on...

I can't find any documentation on how the behavior of copy_tree changes with update=1 set.

Edit nevermind, I looked at the sources and this does exactly what I need it to do. Thanks!

The batteries really are included.

Edit: Edit: It still didn't work, but I think the thumbdrive I was playing around with is bad.

SYSV Fanfic fucked around with this message at 19:05 on Mar 7, 2014

# ? Mar 7, 2014 18:39

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

Plorkyeran posted:

Build wheels for all of your packages?

This seems like the way to go!

Googling about this led me to this nice little article about it: https://rhodecode.com/blog/45/how-we-improved-python-packaging-distribution

# ? Mar 7, 2014 19:46

Michaellaneous: Oct 30, 2013

So, I got the following problem. I always jump into this statement:

code:

...
elif cur_msg[5] != '1' or '2' or '3':
	ircsock.send('PRIVMSG %s Not a correct access level! Please use 1, 2, or 3!\r\n' % (ch))
...

despite the content of cur_msg being 2. What did I do wrong? (Correctly intended, copy pasta kinda hosed up.)

# ? Mar 8, 2014 00:46

salisbury shake: Dec 27, 2011

code:

elif cur_msg[5] != '1' or '2' or '3'

code:

elif (cur_msg[5] != '1') or '2' or '3'

which is always True.

try

code:

elif cur_msg[5] not in '123'

A string is a collection, 'in' tests for membership in a collection, ~~'not' returns the opposite truth value~~ 'not in' is it's own operator

salisbury shake fucked around with this message at 01:48 on Mar 8, 2014

# ? Mar 8, 2014 01:40

John DiFool: Aug 28, 2013

Michaellaneous posted:

So, I got the following problem. I always jump into this statement:
code:
...
elif cur_msg[5] != '1' or '2' or '3':
	ircsock.send('PRIVMSG %s Not a correct access level! Please use 1, 2, or 3!\r\n' % (ch))
...
despite the content of cur_msg being 2. What did I do wrong? (Correctly intended, copy pasta kinda hosed up.)

That's not how the 'or' operator works. In your case '2' is evaluated to true, so that elif is always triggered.

If you want to make sure it's not equal to those constants, then you want something like:

code:

elif cur_msg[5] != '1' and cur_msg[5] != '2' and cur_msg[5] != '3':
	ircsock.send('PRIVMSG %s Not a correct access level! Please use 1, 2, or 3!\r\n' % (ch))

Though if you've already dealt with the 1-3 cases, then you should be using an 'else' clause.

John DiFool fucked around with this message at 01:45 on Mar 8, 2014

# ? Mar 8, 2014 01:42

BigRedDot: Mar 6, 2008

fletcher posted:

Sounds interesting. Not a lot of stars on github though, is it production ready?

Anaconda is at about 40 thousand downloads a month now.

# ? Mar 8, 2014 04:38

fritz: Jul 26, 2003

Cheekio posted:

I want to basically brute force all letter combinations of length n, in order.

But how do I make a generator of variable letter length?

If you're up for the theory, I think there's a big write up of this problem in Knuth 4A.

# ? Mar 8, 2014 06:40

Dominoes: Sep 20, 2007

Looking for wisdom on decorators.

I wrote this one:

Python code:

def retry_if_fail(func):
    def inner(*args):
        result = func(*args)
        if result == "exception":
            return retry(func, *args)
        return result
    return inner


@retry_if_fail
def balances_raw():
    url = 'accounts/{0}/balances.json'.format(MEMBER_NUMBER)
    return exception_handling(url, 'get')

def retry(func, *args):
    time.sleep(20)
    return func(*args)

exception_handling() sets up a http request, checks for various exceptions, and returns "exception" if it finds one. It returns the http response if successful. The decorator's purpose is to simplify code repetition by altering the function's output. If it gets an exception, it waits 20 seconds, and tries again.

For debugging, I set exception handling to always return "exception". balances_raw() is retrying once, then returning "exception". It should keep returning "exception" indefinitely. Any ideas? This is my first attempt at a decorator.

The basic functions work properly; it's my decorator conversion that's messing it up. (Before, I had the retry logic in the balances_raw function.)

Dominoes fucked around with this message at 22:47 on Mar 8, 2014

# ? Mar 8, 2014 21:18

SurgicalOntologist: Jun 17, 2004

The problem might be that the func getting called by retry is the original func, i.e. before it is wrapped by rety_if_fail.

I wonder if also decorating retry would work, or if you should put while logic into retry_if_fail.

E: a third idea is to use

Python code:

return retry(retry_if_fail(func), *args)

SurgicalOntologist fucked around with this message at 22:29 on Mar 8, 2014

# ? Mar 8, 2014 22:26

Dominoes: Sep 20, 2007

Thank you - the third idea works. I think decorating retry is doing an increasing recursion - the first retry goes through one sleep cycle before returning "exception", then two, three four etc.

# ? Mar 8, 2014 22:59

Gmaz: Apr 3, 2011; New DLC for Aoe2 is out: Dynasties of India

Is it good/bad practice to use multiple decorators on a function?

# ? Mar 9, 2014 22:17

Haystack: Jan 23, 2005

Gmaz posted:

Is it good/bad practice to use multiple decorators on a function?

Its perfectly acceptable. Whether or not it's good practice depends on your use case, of course.

# ? Mar 9, 2014 22:40

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

It can make debugging harder. Like all clever features of a programming language, decorators are good when used in moderation.

# ? Mar 10, 2014 01:53

SYSV Fanfic: Sep 9, 2003; by Pragmatica

Trying to get the contents of usercp.php. I tried using urllib and httplib, but for some reason I got caught in a redirect loop. I broke down and used mechanize, which accomplishes the same thing. Can anyone tell me what was wrong with:

code:

import httplib2
import urllib

url='http://forums.somethingawful.com/account.php'
body= {'action':'login', 'username':'keyvin', 'password':'lolnotdumbenoughtocopypaste'}
headers = {'Content-type':'application/x-www-form-urlencoded'}

http = httplib2.Http()

response, content = http.request(url, 'POST', headers=headers, body=urllib.parse.urlencode(body))

url='http://forums.somethingawful.com/usercp.php'
headers = {'Cookie' : response['set-cookie']}

response, content= http.request('url', 'get', headers=headers)

This causes an exception to be thrown about too many redirects. The cookie that gets set by the POST looks to be valid, so I am not sure what is wrong with this code in relation to something awful.

# ? Mar 10, 2014 13:40

NOTinuyasha: Oct 17, 2006; The Great Twist

I have no idea what's wrong with your code, but here's a much cleaner way to login to the forums and make subsequent requests.

NOTinuyasha fucked around with this message at 23:08 on Apr 7, 2014

# ? Mar 10, 2014 14:27

sharktamer: Oct 30, 2011; Shark tamer ridiculous

There's also AwfulPy. I've not had much luck with it in the past though.

notInuyasha, this isn't a python question, but how did you link that highlighted section?

# ? Mar 10, 2014 16:25

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

sharktamer posted:

There's also AwfulPy. I've not had much luck with it in the past though.

notInuyasha, this isn't a python question, but how did you link that highlighted section?

Look at the URL. It ends with #L18-L46

# ? Mar 10, 2014 16:41

NOTinuyasha: Oct 17, 2006; The Great Twist

sharktamer posted:

notInuyasha, this isn't a python question, but how did you link that highlighted section?

Shift-click a line number to highlight one line, or shift-click two line numbers to highlight a range. Then copy the url from your address bar.

# ? Mar 10, 2014 17:45

BigRedDot: Mar 6, 2008

More Bokeh, 0.4.2 this time. New IPython %bokeh magic, more matplotlib and seaborn compatibility and examples, and (finally) windows support for the plot server. Check it out!

# ? Mar 10, 2014 19:55

ahmeni: May 1, 2005; It's one continuous form where hardware and software function in perfect unison, creating a new generation of iPhone that's better by any measure.; Grimey Drawer

BigRedDot posted:

More Bokeh, 0.4.2 this time. New IPython %bokeh magic, more matplotlib and seaborn compatibility and examples, and (finally) windows support for the plot server. Check it out!

Fantastic as always. Any plans for stacked bar charts? I'm after vertical in particular, grouped would be even nicer. We're using highcharts right now and it's okay but very clunky.

# ? Mar 10, 2014 22:46

BigRedDot: Mar 6, 2008

ahmeni posted:

Fantastic as always. Any plans for stacked bar charts? I'm after vertical in particular, grouped would be even nicer. We're using highcharts right now and it's okay but very clunky.

As a matter of fact I very nearly added a 'bokeh.charts' API that would have high level functions for very easily creating schematic plots like bars (grouped, stacked, percentage), histograms, areas (stacked or superimposed), etc. but I wanted to have a wider discussion about just what it should look like. I would expect this API in master in the next couple of weeks and definitely in the next release.

# ? Mar 10, 2014 23:00

sharktamer: Oct 30, 2011; Shark tamer ridiculous

BigRedDot posted:

More Bokeh, 0.4.2 this time. New IPython %bokeh magic, more matplotlib and seaborn compatibility and examples, and (finally) windows support for the plot server. Check it out!

Is this easy enough to use with flask as is? If so, would you be willing to add some use cases to the user guide?

# ? Mar 10, 2014 23:48

ItBurns: Jul 24, 2007

While we're on the topic of charts and such, is Python for Data Analysis worth the read, or am I better off going with (possibly better) books more focused on the language itself? I have a firm grasp on statistics and data analysis, but don't know anything about python. I'd like to learn python to get over some of the 'not a real programming language' hurdles of R, but haven't gotten much further than a couple of online courses.

# ? Mar 11, 2014 01:39

BannedNewbie: Apr 22, 2003; HOW ARE YOU? -> YOSHI?
FINE, THANK YOU. -> YOSHI.

ItBurns posted:

While we're on the topic of charts and such, is Python for Data Analysis worth the read, or am I better off going with (possibly better) books more focused on the language itself? I have a firm grasp on statistics and data analysis, but don't know anything about python. I'd like to learn python to get over some of the 'not a real programming language' hurdles of R, but haven't gotten much further than a couple of online courses.

I have that book and found it to be extremely useful. It basically gives a very nice rundown of the capabilities and syntax of the most essential tools for scientific computing in python (numpy, matplotlib, and pandas) with much of the emphasis on pandas (a library that gives access to somewhat R-like dataframes in python).

I used the book to effectively move most of my data analysis from MATLAB to python, which has proven to be a much more effective way to work ~90% of the time.

# ? Mar 11, 2014 02:53

salisbury shake: Dec 27, 2011

sharktamer posted:

There's also AwfulPy. I've not had much luck with it in the past though.

notInuyasha, this isn't a python question, but how did you link that highlighted section?

what problem did you have with it? I need an excuse to work on it.

# ? Mar 11, 2014 04:19

Adbot: ADBOT LOVES YOU

# ? Jun 2, 2024 17:28

ahmeni: May 1, 2005; It's one continuous form where hardware and software function in perfect unison, creating a new generation of iPhone that's better by any measure.; Grimey Drawer

BigRedDot posted:

As a matter of fact I very nearly added a 'bokeh.charts' API that would have high level functions for very easily creating schematic plots like bars (grouped, stacked, percentage), histograms, areas (stacked or superimposed), etc. but I wanted to have a wider discussion about just what it should look like. I would expect this API in master in the next couple of weeks and definitely in the next release.

Awesome. Looking forward to it.

# ? Mar 11, 2014 04:22

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »