Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

The Gripper: Sep 14, 2004; i am winner

GrumpyDoctor posted:

Is IronPython still being developed?

It is, they generally only release stable builds to coincide with official Python 2.7.x releases though. Since Microsoft abandoned it it's been a lot slower, but work still gets done.

# ? Feb 4, 2013 21:42

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 14:42

BigRedDot: Mar 6, 2008

For anyone that will be (or would like a reason to to be) in the bay area from March 18-20 (right after PyCon), the semi-annual PyData conference that we host will be going on:

http://sv2013.pydata.org

Obviously we'd love to have anyone interested in python and data analysis attend! For everyone that can't make it, all the tutorials and talks will be recorded and and made freely available on Vimeo shortly after the conference. (You can see all the previous talks and tutorials from previous events here.)

# ? Feb 6, 2013 07:26

Ashex: Jun 25, 2007; These pipes are cleeeean!!!

I have a super basic python script for updating the XBMC library, all it does is send the VideoLibrary.Scan command.

I've recently discovered that if the HTTP web server in xbmc hangs up for some reason the script hangs with it never exiting the Scan command. I discovered this when I looked in my SABnzbd queue and found 10 downloads waiting to be processed because the script hadn't exited.

Anyone have recommendations on how to add a timeout to it? I'm doing some reading and all the solutions I've found are simply overly complex. When the web server is functioning correctly the command is nearly instantaneous so I really only need to do a simple "count to 5 then exit" or something.

# ? Feb 6, 2013 08:48

Hefty: Jun 11, 2008

I'm trying to call wpa_supplicant from a python script, but it's putting a space between the path (pwd) and the file name. Any idea why? Or why it's including a path at all?

This is the call:
call(['wpa_supplicant', '-B', '-Dwext', '-i {0}'.format(interface), '-c {0}'.format(config_file_path)])

And this is the error wpa_supp gives:
Failed to read or parse configuration '/home/hefty/scripts/ generated-wpa-supplicant.conf'.

Also, string.format seems like a really stupid way to do those command arguments. Is there a better way?

# ? Feb 6, 2013 09:05

Fleur Bleu: Nov 26, 2006; by Ralp

When I'm typing in secondary prompt and I hit enter, it doesn't give me a new line, but instead executes the thing.
What's up with that?

# ? Feb 6, 2013 13:23

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Ashex posted:

I have a super basic python script for updating the XBMC library, all it does is send the VideoLibrary.Scan command.

I've recently discovered that if the HTTP web server in xbmc hangs up for some reason the script hangs with it never exiting the Scan command. I discovered this when I looked in my SABnzbd queue and found 10 downloads waiting to be processed because the script hadn't exited.

Anyone have recommendations on how to add a timeout to it? I'm doing some reading and all the solutions I've found are simply overly complex. When the web server is functioning correctly the command is nearly instantaneous so I really only need to do a simple "count to 5 then exit" or something.

Start it in a thread and then kill it if it's still going after n seconds?

# ? Feb 6, 2013 14:00

JetsGuy: Sep 17, 2003; science + hockey
=
LASER SKATES

hey guys dumb question regarding overflow.

I want to do some math with **very** big numbers, like 35**200 big etc.

Now, if they were just integers, this wouldn't be a huge deal. Python would just automatically convert them to longs, and woo.

However, I want to be precise here (for example, 35 is really more like 35.3). Sooooo there's the problem. I had thought that stuff like decimal.Decimal would work but it seems to also only be confined to integers.

Is there a way to do math, particularly things like this without just admitting a degree of defeat on the decimal and just making all the numbers integers? This DOES make a difference!!! I was hoping to find a way to force it using a float128 or something.

Thanks!

JetsGuy fucked around with this message at 00:27 on Feb 7, 2013

# ? Feb 7, 2013 00:18

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Use bigfloat? http://packages.python.org/bigfloat/

# ? Feb 7, 2013 00:19

JetsGuy: Sep 17, 2003; science + hockey
=
LASER SKATES

BeefofAges posted:

Use bigfloat? http://packages.python.org/bigfloat/

Hmm. I didn't know about this package, thanks! I was kind of hoping for a built-in solution (although I could certainly use this), does one exist?

# ? Feb 7, 2013 00:23

Scaevolus: Apr 16, 2007

How much precision do you actually want? Do you need 2 decimal points of precision with a number that's 35**200?

GMPY gives you a lot of multiprecision options.

You might find Sage useful as well. It's a comprehensive Python-based math system.

# ? Feb 7, 2013 00:28

JetsGuy: Sep 17, 2003; science + hockey
=
LASER SKATES

Scaevolus posted:

How much precision do you actually want? Do you need 2 decimal points of precision with a number that's 35**200?

GMPY gives you a lot of multiprecision options.

You might find Sage useful as well. It's a comprehensive Python-based math system.

Perhaps precision isn't the right word here. And maybe even more to the point, I maybe mis-stated my question to begin with. I don't really care about the decimals precision so much as being able to do the operations to begin with.

For example:

code:

In [63]: 35**199.
Out[63]: 1.8601194148960557e+307

In [64]: 35.1**199.
Out[64]: 3.281830725541065e+307

In [65]: 35.5**199.
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
/Users/JetsGuy/<ipython-input-65-ac7388840616> in <module>()
----> 1 35.5**199.

OverflowError: (34, 'Result too large')

In [66]: 35**200
Out[66]: 651041795213619463956247586936077487434111469539575300902579176560
462999713515884487479514344544881943486453262894322315746363855567874310852
103171536994281209881245266837821479680338359634819260697214819508504812412
290509792493528193158691118879388019630006823688776231018326967614484601654
112339019775390625L

In [67]: 35**200.
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
/Users/JetsGuy/<ipython-input-67-213507bc0b99> in <module>()
----> 1 35**200.

OverflowError: (34, 'Result too large')

So I'm literally breaking the OverFlow limits. I've tried going to 128-bit floats, and it's not helping. :smith:

EDIT:
Holy breakage Batman!

JetsGuy fucked around with this message at 00:38 on Feb 7, 2013

# ? Feb 7, 2013 00:35

Houston Rockets: Apr 15, 2006

Hefty posted:

... it's putting a space between the path (pwd) and the file name. Any idea why? Or why it's including a path at all?

How did you assign config_file_path?

# ? Feb 7, 2013 00:40

The Gripper: Sep 14, 2004; i am winner

JetsGuy posted:

Perhaps precision isn't the right word here. And maybe even more to the point, I maybe mis-stated my question to begin with. I don't really care about the decimals precision so much as being able to do the operations to begin with.

I suppose you could use fractions.Fraction, which is for rational number arithmetic.

Python code:

>>> x = Fraction(35.5)
>>> x**200
Fraction(1785129569346076924196985267449190664862498632783514540772889843827458524312117666282676970
5109222981688420093214109207896131946422896731843680197632286678426362811714986668716603139157898035
7224390713091435934404588027191142494665541761058627763579220579563576028137791954173890984960196599
96794857397552444264801724688719583543715407252300922319918504277557927671224001, 160693804425899027
5541962092341162602522202993782792835301376)

# ? Feb 7, 2013 00:44

JetsGuy: Sep 17, 2003; science + hockey
=
LASER SKATES

The Gripper posted:

I suppose you could use fractions.Fraction, which is for rational number arithmetic.

Python code:

>>> x = Fraction(35.5)
>>> x**200
Fraction(1785129569346076924196985267449190664862498632783514540772889843827458524312117666282676970
5109222981688420093214109207896131946422896731843680197632286678426362811714986668716603139157898035
7224390713091435934404588027191142494665541761058627763579220579563576028137791954173890984960196599
96794857397552444264801724688719583543715407252300922319918504277557927671224001, 160693804425899027
5541962092341162602522202993782792835301376)

I actually came up with a solution that this is kinda doing on my drive home. Excellent. I was thinking of doing something like:

code:

In [9]: 35.123**200
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
/Users/JetsGuy/<ipython-input-9-d94a011aa08b> in <module>()
----> 1 35.123**200

OverflowError: (34, 'Result too large')

In [10]: 35**200 + (123**200)/(1000*200)
Out[10]: 47862159775277164454893607233692734869173391984716909108540594362194803443869
07148424076561025754194712038115852249299325894510285806546307904470139260175642519075
2747901414124589223457120544889762397750106831302100375997151695190866251670612660546
289038772439957366422033659469345815347293718554301610905482154494813987599782875897742
717900593495971296675426287326019527146029194163922555184822966416930685664967L

Fraction may make life easier...

I'd also have to do a similiar treatment if the power was a decimal like 200.1... ugh.

EDIT:
I would have expected something like this to exist in the built-ins. It's an obvious thing that would be needed!! :psyduck:

JetsGuy fucked around with this message at 01:28 on Feb 7, 2013

# ? Feb 7, 2013 01:19

The Gripper: Sep 14, 2004; i am winner

Completely unrelated to the current discussion but I only just found out that _ represents the last value that wasn't assigned to a variable in the Python console.

All these years I'd never known, this would have saved a lot of time!

# ? Feb 7, 2013 01:27

Hefty: Jun 11, 2008

Houston Rockets posted:

How did you assign config_file_path?

I'm just setting it to the file name.
config_file_path = 'generated-wpa-supplicant.conf'

It's gotta be the call() method that's doing it. I just tried appending the current working directory, thinking maybe it wants an absolute path:

code:

call(['wpa_supplicant', '-B', '-Dwext', '-i {0}'.format(interface), '-c {0}/{1}'.format(os.getcwd(),config_file_path)])

But now it's doing this:
Failed to read or parse configuration '/home/hefty/scripts/ /home/hefty/scripts/generated-wpa-supplicant.conf'.

# ? Feb 7, 2013 02:33

Houston Rockets: Apr 15, 2006

Hefty posted:

It's gotta be the call() method that's doing it. I just tried appending the current working directory, thinking maybe it wants an absolute path:

Try popen with the cwd keyword argument.

# ? Feb 7, 2013 03:58

Hefty: Jun 11, 2008

Houston Rockets posted:

Try popen with the cwd keyword argument.

Sorry if this is really basic stuff, I just started learning python.
So do you mean like this? I hard-coded the current directory just to see if it works.

code:

command = ['wpa_supplicant', '-B', '-Dwext', '-i {0}'.format(interface), '-c {0}'.format(config_file_path)]
subprocess.Popen(command, cwd='/home/hefty/scripts/')

Still gives me the same output though.
Failed to read or parse configuration '/home/hefty/scripts/ generated-wpa-supplicant.conf'.

I tried passing all the command args to Popen individually, but it complained that 'TypeError: bufsize must be an integer'.

# ? Feb 7, 2013 04:42

Emacs Headroom: Aug 2, 2003

JetsGuy posted:

hey guys dumb question regarding overflow.

I want to do some math with **very** big numbers, like 35**200 big etc.

Can you change everything to log scale and just do log operations (so * is + etc.)?

# ? Feb 7, 2013 04:47

breaks: May 12, 2001

Hefty posted:

Sorry if this is really basic stuff, I just started learning python.
So do you mean like this? I hard-coded the current directory just to see if it works.
code:
command = ['wpa_supplicant', '-B', '-Dwext', '-i {0}'.format(interface), '-c {0}'.format(config_file_path)]
subprocess.Popen(command, cwd='/home/hefty/scripts/')
Still gives me the same output though.
Failed to read or parse configuration '/home/hefty/scripts/ generated-wpa-supplicant.conf'.

Oh to be able to bold a space.

I would bet some money at this stage that the problem is with wpa_supplicant and that '-c{0}' will fix it.

# ? Feb 7, 2013 07:40

Hefty: Jun 11, 2008

breaks posted:

I would bet some money at this stage that the problem is with wpa_supplicant and that '-c{0}' will fix it.

Welp, that was it :doh:

. Thanks guys!

# ? Feb 7, 2013 08:45

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Instead of directly using subprocess, I suggest using a wrapper like sarge or envoy. subprocess is really annoying to work with directly.

# ? Feb 7, 2013 10:32

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

BeefofAges posted:

Instead of directly using subprocess, I suggest using a wrapper like sarge or envoy. subprocess is really annoying to work with directly.

Seems like it could be vulnerable to shell injections. I don't see what's that bad about subprocess, other than having to know how arguments work.

# ? Feb 7, 2013 10:46

Hefty: Jun 11, 2008

Suspicious Dish posted:

Seems like it could be vulnerable to shell injections. I don't see what's that bad about subprocess, other than having to know how arguments work.

Hmm, google'ing envoy turns up this example: envoy.run('rm -v %s' % item)
Which could presumably turn into: envoy.run('rm -v %s' % '-f /')

Not that anyone is using this stupid little script I'm writing, but I'll probably stick with parameterized arguments since I don't know what the hell I'm doing.

# ? Feb 7, 2013 12:20

tef: May 30, 2004; -> some l-system crap ->

Suspicious Dish posted:

Seems like it could be vulnerable to shell injections. I don't see what's that bad about subprocess, other than having to know how arguments work.

iirc Reading from stdin or stderr can block the other. It's easier to spawn off threads to collect them.

# ? Feb 7, 2013 14:56

JetsGuy: Sep 17, 2003; science + hockey
=
LASER SKATES

Hey y'all, I tried using bigfloat, but there was some library issue. I had installed mpfr and that other thing no problem, but it was still complaining.

So I instead took the route of evaluating the function in log space like was suggested before. This helps evaluate, but ultimately, when trying to convert back to linear space.

For further clarification, I'm trying to evaluate the Poisson distribution at large numbers.

Other problems I'm seeing is that virtually all the factorial/gamma functions I am finding limit themselves to double precision. The problem is definitely in the overflow and not in the precision, so sorry about the earlier confusion.

EDIT:
I hate to say this, but this would be easier in MATLAB, which a colleague of mine is using and having no problems. I guess MATLAB automatically has some kind of built in quadruple precision avail if need be. :smith:

JetsGuy fucked around with this message at 22:38 on Feb 7, 2013

# ? Feb 7, 2013 22:35

JetsGuy: Sep 17, 2003; science + hockey
=
LASER SKATES

Welp, I have now thoroughly embarrassed myself before the entire thread. :smith:

I swear, I googled for this before, and only could find RNGs.... Nonetheless, after all this time looking for how to handle huge floats, after wasting colleagues time looking for answers as to how best to "trick" the system... I find this:

code:

In [233]: from scipy.stats import poisson

In [234]: poisson.pmf(200., 35.)
Out[234]: 5.2049116662890937e-82

Please, someone just shoot me, for I am a giant loving idiot.

I still am kind of interested in how that is calculated though, seeing as python/scipy seem to be so vehemently against doing any kind of long-float operations, as I've now seen.

JetsGuy fucked around with this message at 00:00 on Feb 8, 2013

# ? Feb 7, 2013 23:57

Emacs Headroom: Aug 2, 2003

JetsGuy posted:

Hey y'all, I tried using bigfloat, but there was some library issue. I had installed mpfr and that other thing no problem, but it was still complaining.

So I instead took the route of evaluating the function in log space like was suggested before. This helps evaluate, but ultimately, when trying to convert back to linear space.

For further clarification, I'm trying to evaluate the Poisson distribution at large numbers.

If you're doing anything like evaluating likelihoods of Poisson random variables, you know you can rescale the log-likelihoods before you convert them back (since you're going to be normalizing to get probabilities anyway).

E.g, you have

Python code:

llh_vector = function_that_gives_numpy_array_of_log_likelihoods()
llh_vector -= llh_vector.max() #let's normalize everything to around 0 to avoid precision problems
lh_vector = exp(llh_vector)
p_vector = lh_vector / lh_vector.sum() #now you have probabilities assuming uniform Bayesian prior

# ? Feb 8, 2013 00:50

Ireland Sucks: May 16, 2004

Long shot here, but I'm occasionally doing a lot of blitting in pygame and tried to just run the function doing it in a new process but that didn't work because a separate process doesn't share the same display and I just get 'pygame.error: display Surface quit' errors. I considered using threads but from what I can see they are likely to have the same problem and pygames builtin thread support gives me a bunch of errors when I call threads.init() it and probably hasn't been ported to the Python 3 version.

Is there a way around this or is loading some graphics in the background too much to ask?

# ? Feb 8, 2013 02:38

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Suspicious Dish posted:

Seems like it could be vulnerable to shell injections. I don't see what's that bad about subprocess, other than having to know how arguments work.

The syntax is overly complex, given that all I usually want is "run this in the command line and give me the output".

# ? Feb 8, 2013 05:14

BigRedDot: Mar 6, 2008

JetsGuy posted:

I still am kind of interested in how that is calculated though, seeing as python/scipy seem to be so vehemently against doing any kind of long-float operations, as I've now seen.

They work with the log:

code:

    def _logpmf(self, k, mu):
        Pk = k*log(mu)-gamln(k+1) - mu
        return Pk
    def _pmf(self, k, mu):
        return exp(self._logpmf(k, mu))

And if you really care, log(gamma(a)) is implemented:

https://github.com/scipy/scipy/blob/master/scipy/special/cdflib/gamln.f
https://github.com/scipy/scipy/blob/master/scipy/special/cdflib/gamln1.f

# ? Feb 8, 2013 06:19

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

BeefofAges posted:

The syntax is overly complex, given that all I usually want is "run this in the command line and give me the output".

Python code:

import subprocess
output = subprocess.check_output(['fart', '--boners'])

# ? Feb 8, 2013 06:36

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Suspicious Dish posted:

Python code:

import subprocess
output = subprocess.check_output(['fart', '--boners'])

I'll post some examples tomorrow when I'm in the office.

# ? Feb 8, 2013 08:09

Kosani: Jun 30, 2004; Ni ni.

Is there a way I can make all instances of a class share specific properties?

Here is a pseudocode example of what I mean:

code:

tigerA = Tiger()
tigerB = Tiger()

print(tigerA.pawSize)
print(tigerB.pawSize)

>> 5
>> 5

code:

tigerB.pawSize = 20
print(tigerA.pawSize)
print(tigerB.pawSize)

>> 20
>> 20

I'm almost certain that I read it was easy to do this but I can't find it on google because I don't know what the concept is called. If anyone knows and could tell me I'd very much appreciate it, thank you. I thought it was a matter of using, for example, 'Tiger' instead of 'self' when defining class properties but that doesn't appear to work.

Kosani fucked around with this message at 18:55 on Feb 9, 2013

# ? Feb 9, 2013 18:46

how!!: Nov 19, 2011; by angerbot

You do that by setting the variable on the Tiger class. You define "paw_size" (don't use camel case, its not python style) on the Tiger class:

code:


class Tiger(object):
    paw_size = 20 # belongs to the class, but accessible from any instance

>>> ta = Tiger()
>>> tb = Tiger()
>>> ta.paw_size
20
>>> tb.paw_size
20
>>> Tiger.paw_size = 500
>>> ta.paw_size
500
>>> tb.paw_size
500

# ? Feb 9, 2013 19:08

Kosani: Jun 30, 2004; Ni ni.

how!! posted:

You do that by setting the variable on the Tiger class. You define "paw_size" (don't use camel case, its not python style) on the Tiger class:
code:
class Tiger(object):
    paw_size = 20 # belongs to the class, but accessible from any instance

>>> ta = Tiger()
>>> tb = Tiger()
>>> ta.paw_size
20
>>> tb.paw_size
20
>>> Tiger.paw_size = 500
>>> ta.paw_size
500
>>> tb.paw_size
500

Thank you for getting back to me so quickly. Can you explain why you inherit from object when defining the class? For me, your code works either way (using 'class Tiger(object):' or class Tiger: ') Is that just good practice for some reason? I'm new to python.

Kosani fucked around with this message at 19:46 on Feb 9, 2013

# ? Feb 9, 2013 19:41

Rothon: Jan 4, 2012

Kosani posted:

Thank you for getting back to me so quickly. Can you explain why you inherit from object when defining the class? For me, your code works either way (using 'class Tiger(object):' or class Tiger: ') Is that just good practice for some reason? I'm new to python.

"New-style classes" inherit from object: http://www.python.org/doc/newstyle/

# ? Feb 9, 2013 19:49

No Safe Word: Feb 26, 2005

Rothon posted:

"New-style classes" inherit from object: http://www.python.org/doc/newstyle/

* except in Python 3.x where you don't have to do that.

# ? Feb 9, 2013 19:57

QuarkJets: Sep 8, 2008

In Python 2.6 I want to take a dictionary and convert all of the keys, which are strings with various cases, to lowercase strings. Right now I am doing it like this:

code:


dict_low = dict((k.lower(), v) for k, v in olddict.items())

I just want to change the keys to lowercase, not create a second copy of the dictionary in memory. Does this code do that, and how can I check that this is the case?

# ? Feb 11, 2013 19:42

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 14:42

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

QuarkJets posted:

In Python 2.6 I want to take a dictionary and convert all of the keys, which are strings with various cases, to lowercase strings. Right now I am doing it like this:
code:
dict_low = dict((k.lower(), v) for k, v in olddict.items()) 
I just want to change the keys to lowercase, not create a second copy of the dictionary in memory. Does this code do that, and how can I check that this is the case?

I'm pretty sure that immediately after that line both the old and new dictionaries are in memory in full, because olddict isn't being modified and is still in scope. Given you want to overwrite the old dictionary as you go, you might have to do something like this (although there might well be a builtin I'm not aware of that does the same thing)

code:

for k, v in mydict.items():
    lowerk = k.lower()
    if k != lowerk:
        mydict[lowerk] = v
        del mydict[k]

# ? Feb 11, 2013 19:52

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »