Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

QuarkJets: Sep 8, 2008

Have you tried reading the documentation?

https://pythonhosted.org/joblib/parallel.html#common-usage

According to that, the function that you want to parallelize needs to be a generator. It sounds like you need to modify your code

# ? Apr 26, 2015 18:33

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 14:35

Nippashish: Nov 2, 2005; Let me see you dance!

Cingulate posted:

f is a fairly long, complicated and badly written function so I don't want to bother anybody with it.
When I run it outside of the parallel loop, it works.

It looks to me like you're using joblib correctly. Can you trigger the same weirdness with a different f? Does f interact with the outside world at all?

# ? Apr 26, 2015 19:13

Cingulate: Oct 23, 2012; by Fluffdaddy

QuarkJets posted:

Have you tried reading the documentation?

https://pythonhosted.org/joblib/parallel.html#common-usage

According to that, the function that you want to parallelize needs to be a generator. It sounds like you need to modify your code

In what I did, I tried to copy the docs. From what I understand, we write the code as a generator - i.e., my (delayed(f)(item) for item in list_of_items), where f is a "regular" function that returns something.

FWIW it works if I set n_jobs to -2 ...

Actually, you pointing out that (delayed(f)(item) for item in list_of_items) is a generator finally cleared up the joblib syntax for me.

# ? Apr 26, 2015 19:16

Cingulate: Oct 23, 2012; by Fluffdaddy

Nippashish posted:

It looks to me like you're using joblib correctly. Can you trigger the same weirdness with a different f? Does f interact with the outside world at all?

I don't see how f would have any side effects.
I do do random.seed(an_int) in there. This is pointless, as I'm just realizing, as an_int is set to the same value for all iterations by my usage of partial() to wrap f, but that's the only thing I can see loving with globals.
I'm accessing a bunch of globals however.

But the only stuff that's changed is created inside the function, and the function returns to a new list.

I'm possibly missing something very obvious, I'm a terrible programmer.

I haven't found another f it does this for, though I haven't tried much.

# ? Apr 26, 2015 19:20

QuarkJets: Sep 8, 2008

I'm almost certain that it's something weird due to your function, so I'd like to suggest Nippashish's advice as well: try a very simple, working version of f that parallelizes correctly and keep building up f until the parallelization breaks. This is a pretty effective debugging technique

Cingulate posted:

I'm accessing a bunch of globals however.

This could also be the culprit, I would strongly encourage you to not even use globals. Ever.

# ? Apr 27, 2015 17:54

hooah: Feb 6, 2006; WTF?

Something cool I just found out about Python: if you're unpickling e.g. a Counter object from a file, you don't need to do from collections import Counter. That's pretty nifty!

# ? Apr 28, 2015 14:39

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

pickle tries really hard to store the full module path in the file itself, so it knows where to import it from afterwards.

# ? Apr 28, 2015 16:23

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

How hard is it to get patches to the python standard library accepted? I've fixed a few things in asyncio, but they're kinda edge cases. I don't really want to spend a bunch of time on shepherding them through a long drawn out process...

# ? Apr 28, 2015 18:36

IAmKale: Jun 7, 2007; やらないか; Fun Shoe

I'm cross-posting from the Django thread because I think this might also be a general Python question.

Is it possible to pass a variable assignment statement as a parameter when I make a method call? For example:

Python code:

# often-repeated code used in many places
if is_admin:
	specific_func_call(always_changes=value)
elif is_manager:
	specific_fun_call(this_changed=other_val)

In this case, I'm hoping it's possible to pass always_changes=value or this_changed=other_val as parameters to a method like this:

Python code:

# DRY replacement function
def custom_function(admin_statement, manager_statement):
	if is_admin:
		specific_func_call(admin_statement)
	elif is_manager:
		specific_func_call(manager_statement)

and then call custom_function() like so:

Python code:

custom_function(
	always_changes=value,
	this_changed=other_val)

Is such a thing possible in Python? I'd need it to be able to handle basically anything - always_changes and this_changed could potentially be called anything.

# ? Apr 28, 2015 20:48

Nippashish: Nov 2, 2005; Let me see you dance!

You can expand a dictionary into keyword arguments like this:

code:

def my_function(a, b):
    print(a, b)

my_dict = { "a": "hello", "b": "there"}
my_function(**my_dict)

You can also capture arbitrary keyword arguments in a dictionary like this:

code:

def my_other_function(stuff, **kwargs):
    print(stuff)
    print(kwargs)

my_other_function("this is stuff", bob="bob", dole="dole")

And of course you can combine both of these things like this:

code:

def my_third_function(**kwargs):
    my_function(**kwargs)

my_third_function(a=1, b=2)

# ? Apr 28, 2015 21:20

IAmKale: Jun 7, 2007; やらないか; Fun Shoe

Nippashish posted:

And of course you can combine both of these things like this:
code:
def my_third_function(**kwargs):
    my_function(**kwargs)

my_third_function(a=1, b=2)

Thank you for the pointers. I ended up with something like this:

Python code:

def custom_func(model, is_admin, *args, **kwargs):
    if is_admin:
        args = kwargs['admin']
    else:
        args = kwargs['manager']

    model.func_call(**args)

custom_func(
    SomeObject,
    True,
    admin={'location__id': 2},
    manager={'employee__location__id': 4}
)

It's a little too convoluted for my tastes - I had to get creative with my **kwargs because I have no way of knowing what they might be ahead of time. It seems to work upon testing, though, so it'll do.

IAmKale fucked around with this message at 22:06 on Apr 28, 2015

# ? Apr 28, 2015 22:04

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

Thermopyle posted:

How hard is it to get patches to the python standard library accepted? I've fixed a few things in asyncio, but they're kinda edge cases. I don't really want to spend a bunch of time on shepherding them through a long drawn out process...

It's been mostly painless for me, given my track record of "a patch to the zipfile module" and "a patch to the base64 module"

# ? Apr 29, 2015 00:41

ShadowHawk: Jun 25, 2000; CERTIFIED PRE OWNED TESLA OWNER

What if I wanted to contribute something that's only useful in unit tests?

# ? Apr 29, 2015 04:48

MorsAnima: Nov 29, 2010; I just feel completely shit on. These changes won't make a cat in hell's difference.

Is this a half decent tutorial for machine learning? The book that my tutor recommended ("Building Machine Learning Systems with Python") has been getting pretty poor reviews and comments about poorly-explained code, buggy examples and whatever else, so I've taken it on myself to find something else for now.

# ? Apr 29, 2015 22:53

Cingulate: Oct 23, 2012; by Fluffdaddy

Is there any reason for not using scikit-learn instead?

# ? Apr 29, 2015 23:11

Nippashish: Nov 2, 2005; Let me see you dance!

MorsAnima posted:

Is this a half decent tutorial for machine learning? The book that my tutor recommended ("Building Machine Learning Systems with Python") has been getting pretty poor reviews and comments about poorly-explained code, buggy examples and whatever else, so I've taken it on myself to find something else for now.

Are you trying to learn python or to learn machine learning or to learn how to do machine learning in python? These are three different things and the sources you'd use for each of them are different.

That set of tutorials looks like it covers a lot of very basic python things in a very superficial way.

# ? Apr 29, 2015 23:13

hooah: Feb 6, 2006; WTF?

I'm trying to evaluate a program I wrote by comparing predicted lines against ground-truth lines. How can I pull a line from each file? I tried doing for ground_truth_line, pred_line in ground_truth, predicted: where ground_truth and predicted are the respective files, but I got an error "too many values to unpack".

# ? Apr 30, 2015 15:21

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

hooah posted:

I'm trying to evaluate a program I wrote by comparing predicted lines against ground-truth lines. How can I pull a line from each file? I tried doing for ground_truth_line, pred_line in ground_truth, predicted: where ground_truth and predicted are the respective files, but I got an error "too many values to unpack".

I think you want

Python code:

from itertools import izip

for ground_truth_line, pred_line in izip(ground_truth, predicted):
   #etc

# ? Apr 30, 2015 15:35

hooah: Feb 6, 2006; WTF?

Munkeymon posted:

I think you want

Python code:

from itertools import izip

for ground_truth_line, pred_line in izip(ground_truth, predicted):
   #etc

"Cannot import name 'izip'"

I'm using PyCharm with Python 3.4.1.

# ? Apr 30, 2015 15:46

KICK BAMA KICK: Mar 2, 2009

hooah posted:

"Cannot import name 'izip'" I'm using PyCharm with Python 3.4.1.

In Python 3 just use the built-in zip. As I recall the difference is that Python 2's zip has the limitation that it needs actual sequences like lists or tuples rather than iterators, which izip was invented to handle. In Python 3 they just made zip capable of handling iterators and got rid of izip.

# ? Apr 30, 2015 15:53

EAT THE EGGS RICOLA: May 29, 2008

hooah posted:

"Cannot import name 'izip'" I'm using PyCharm with Python 3.4.1.

just use zip in python 3

# ? Apr 30, 2015 16:11

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

hooah posted:

"Cannot import name 'izip'" I'm using PyCharm with Python 3.4.1.

My bad - I forgot I was living in the future now

KICK BAMA KICK posted:

In Python 3 just use the built-in zip. As I recall the difference is that Python 2's zip has the limitation that it needs actual sequences like lists or tuples rather than iterators, which izip was invented to handle. In Python 3 they just made zip capable of handling iterators and got rid of izip.

zip in 2.x handles iterators just fine but it consumes them in their entirety immediately in order to return a list as opposed to izip which returns an iterator that consumes the input iterators as needed, so zip in 2.x doesn't actually do what hooah was trying to do which is iterate over the lines in the files until there was a difference.

# ? Apr 30, 2015 16:16

hooah: Feb 6, 2006; WTF?

Yup, zip works great. Thanks.

# ? Apr 30, 2015 21:25

MorsAnima: Nov 29, 2010; I just feel completely shit on. These changes won't make a cat in hell's difference.

Nippashish posted:

Are you trying to learn python or to learn machine learning or to learn how to do machine learning in python? These are three different things and the sources you'd use for each of them are different.

That set of tutorials looks like it covers a lot of very basic python things in a very superficial way.

Machine Learning in Python. I'm already fairly experienced with Python, and only mildly so in machine learning. Mix the two and improve at both, ideally. You're right about that tutorial, it does cover a lot of basic stuff - most of which I just gloss over, I don't need it.

I'll work through it, I doubt it'll take too long.

# ? May 1, 2015 16:14

Nippashish: Nov 2, 2005; Let me see you dance!

MorsAnima posted:

Machine Learning in Python. I'm already fairly experienced with Python, and only mildly so in machine learning. Mix the two and improve at both, ideally. You're right about that tutorial, it does cover a lot of basic stuff - most of which I just gloss over, I don't need it.

I'll work through it, I doubt it'll take too long.

In that case the scikit-learn docs should be more than enough to get you up and running. Scikit-learn is an extensive and high quality library, and their documentation is really good.

# ? May 1, 2015 19:31

unixbeard: Dec 29, 2004

MorsAnima posted:

Machine Learning in Python. I'm already fairly experienced with Python, and only mildly so in machine learning. Mix the two and improve at both, ideally. You're right about that tutorial, it does cover a lot of basic stuff - most of which I just gloss over, I don't need it.

I'll work through it, I doubt it'll take too long.

Machine Learning in Action and Machine Learning An Algorithmic Perspective and both useful books that use python.

# ? May 2, 2015 01:47

underage at the vape shop: May 11, 2011; by Cyrano4747

here is some code we got given that we can't change for an assignment:

code:

class Station(object):
    """A class for storing yearly average temperature data for a given station
    """
    def __init__(self, stationfile):
        """ Constructor: Station(str)"""
        self._data = load_data_points(stationfile)
        keys = self._data.keys()
        self._min_year = min(keys)
        self._max_year = max(keys)
        temps = self._data.values()
        self._min_temp = min(temps)
        self._max_temp = max(temps)
        base = os.path.basename(stationfile)
        if not base.endswith('.txt'):
            raise(FileExtensionException())
        self._name = base.replace(".txt", "")

Why can't I access the data variable?

code:

class TemperatureData(Station):

    def printsomething(self):
        print (self._data)

code:

    def load_file(self):
        filepath= filedialog.askopenfilename(filetypes = (("Text files", "*.txt"), ))
        a = filepath.rpartition('/')
        filename=a[2]
        
        TemperatureData.printsomething(filename)

That code (when you call load_file with a button on the screen) gives:

code:

Exception in Tkinter callback
Traceback (most recent call last):
  File "A:\Programs\Python\lib\tkinter\__init__.py", line 1533, in __call__
    return self.func(*args)
  File "A:\Programs\Python\assignments\assign2.py", line 56, in load_file
    TemperatureData.printsomething(filename)
  File "A:\Programs\Python\assignments\assign2.py", line 25, in printsomething
    print (self._data)
AttributeError: 'str' object has no attribute '_data'

underage at the vape shop fucked around with this message at 16:43 on May 4, 2015

# ? May 4, 2015 16:40

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

It should be self.printsomething(filename)

# ? May 4, 2015 16:42

underage at the vape shop: May 11, 2011; by Cyrano4747

Suspicious Dish posted:

It should be self.printsomething(filename)

i wasn't clear. The second function is in a different class, the one that controls the ui.

# ? May 4, 2015 16:44

KICK BAMA KICK: Mar 2, 2009

Suspicious Dish posted:

It should be self.printsomething(filename)

printsomething(self) doesn't look like it takes an argument, so I'm not even sure what's supposed to be happening there. Maybe load_file is supposed to be a class method, a factory for TemperatureData instances, upon which you would then call the printsomething method?

e: ^^^ OK, yeah, then what you need to do in that method is instantiate a TemperatureData, passing filename to its constructor, and then, if you want, call the instance method printsomething.

Python code:

temp_data = TemperatureData(filename)
temp_data.printsomething()

KICK BAMA KICK fucked around with this message at 16:48 on May 4, 2015

# ? May 4, 2015 16:46

underage at the vape shop: May 11, 2011; by Cyrano4747

KICK BAMA KICK posted:

printsomething(self) doesn't look like it takes an argument, so I'm not even sure what's supposed to be happening there. Maybe load_file is supposed to be a class method, a factory for TemperatureData instances, upon which you would then call the printsomething method?

e: ^^^ OK, yeah, then what you need to do in that method is instantiate a TemperatureData, passing filename to its constructor, and then, if you want, call the instance method printsomething.
Python code:
temp_data = TemperatureData(filename)
temp_data.printsomething()

Yay this worked. This assignment really sucks rear end, it's not clear what it wants and the lecture notes aren't very helpful at all.

Thanks.

# ? May 4, 2015 16:51

underage at the vape shop: May 11, 2011; by Cyrano4747

I've been stumped on this next part for couple of hours, I actually started with this before moving on and doing the other parts but I've done as much as I can without this. The last question was actually building into this. Sorry if this is too much and I'll understand if you want me to go do my own assignments but this is the description of what we are supposed to do for this class:

This is my code so far:

I really have no idea how or what it wants me to actually do for the get data and get stations bit. I really have no idea how to do that with only self as an argument for both.

In the example bit it has example inputs and outputs, this is what it says relating to those 2 methods

code:

>>> data = TemperatureData()
>>> data.load_data(�Brisbane.txt�)
>>> data.get_data()
{�Brisbane�: Station(Brisbane)}
>>> data.get_stations()
[�Brisbane�]
>>> data.get_ranges()
(1950, 2014, 24.267, 25.947)
>>> data.load_data(�Adelaide.txt�)
>>> data.get_stations()
[�Brisbane�, �Adelaide�]
>>> data.get_data()
{�Brisbane�: Station(Brisbane), �Adelaide�: Station(Adelaide)}

Where do I even start here? I'm absolutely clueless. The other parts of this class I can do, I know what it wants from me if i don't know the exact code yet, but this part is just ?????????????????

# ? May 4, 2015 17:24

Space Kablooey: May 6, 2009

To me it looks like you have to load your station objects on a dictionary that is a member of the class TemperatureData.

Your get_data should be simple enough when you get to that, and if you are stumped on get_stations, you can call a function called dir on that dictionary, and that should give you some ideas.

# ? May 4, 2015 17:49

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

HardDisk posted:

Your get_data should be simple enough when you get to that, and if you are stumped on get_stations, you can call a function called dir on that dictionary, and that should give you some ideas.

That one actually mentions "in the order they were loaded", so you have to store them in a separate list, or use an OrderedDictionary. That is a pretty bullshit assignment.

Save get_stations for last, then we can help you with that one.

# ? May 4, 2015 17:51

Space Kablooey: May 6, 2009

I'm the horror because I didn't even read the assignment :smith:

Also, check the indentation of the __init__ method of the TemperaturePlotApp.

# ? May 4, 2015 17:59

underage at the vape shop: May 11, 2011; by Cyrano4747

Fixed that indentation.

I'm not sure how I make the dictionary like they want, where get data will return that dictionary for all instances of the class. Do I make a blank dictionary in __init__ and have get data add to it?

# ? May 5, 2015 04:11

underage at the vape shop: May 11, 2011; by Cyrano4747

I can't figure this out in a way that keeps the methods in TemperatureData the way they described.

If I make it inherit from Station (I showed station Init a few posts up) I can use the variables defined in its init to do my get_data. However, this means that:

code:

temp_data=TemperatureData()

will no longer work because Station has an extra argument in it's init, and I need it to work. I can't figure out how to do it without inheriting and when I do inherit, I can't figure out how to make it do this:

code:

>>> data = TemperatureData()
>>> data.load_data(�Brisbane.txt�)
>>> data.get_data()
{�Brisbane�: Station(Brisbane)}
>>> data.get_stations()
[�Brisbane�]
>>> data.get_ranges()
(1950, 2014, 24.267, 25.947)
>>> data.load_data(�Adelaide.txt�)
>>> data.get_stations()
[�Brisbane�, �Adelaide�]
>>> data.get_data()
{�Brisbane�: Station(Brisbane), �Adelaide�: Station(Adelaide)}

This assignment is way harder than the last one. This is such bullshit I'm so mad at it, I've been stuck on this for hours and hours.

# ? May 5, 2015 06:01

QuarkJets: Sep 8, 2008

A Saucy Bratwurst posted:

I can't figure this out in a way that keeps the methods in TemperatureData the way they described.

If I make it inherit from Station (I showed station Init a few posts up) I can use the variables defined in its init to do my get_data. However, this means that:
code:
temp_data=TemperatureData()
        
will no longer work because Station has an extra argument in it's init, and I need it to work. I can't figure out how to do it without inheriting and when I do inherit, I can't figure out how to make it do this:
code:
>>> data = TemperatureData()
>>> data.load_data(�Brisbane.txt�)
>>> data.get_data()
{�Brisbane�: Station(Brisbane)}
>>> data.get_stations()
[�Brisbane�]
>>> data.get_ranges()
(1950, 2014, 24.267, 25.947)
>>> data.load_data(�Adelaide.txt�)
>>> data.get_stations()
[�Brisbane�, �Adelaide�]
>>> data.get_data()
{�Brisbane�: Station(Brisbane), �Adelaide�: Station(Adelaide)}
This assignment is way harder than the last one. This is such bullshit I'm so mad at it, I've been stuck on this for hours and hours.

1) load_data reads a file and puts data from that file into a dictionary.
2) get_data returns the entire dictionary
3) get_stations returns a list of stations that were inserted into the dictionary, in the order in which they were inserted

So what does TemperatureData need to have?

1) a dictionary where the keys are station names and the values are Station objects
2) a list of station names

Here's a constructor that contains these things:

code:

def TemperatureData():
  def __init__(self):
    self.data_dict = {}
    self.station_list = []

That's a starting point. From there, you can create a bunch of methods that fill self.data_dict and self.station_list.

# ? May 5, 2015 07:09

underage at the vape shop: May 11, 2011; by Cyrano4747

Thanks that works almost 100%.

Typing out the way you look at making the classes actually helps a lot too, thats a good thought process rather than me making methods and trying to get init to work with that.

Now i just need to figure out how to have the dictionary save across every instance of TemperatureData (and then do the same for the list) so it does the thing where it has every station that's been loaded so far.

e: got it, but now theres nothing at all inside of __init__ but I don't think thats an issue is it?

underage at the vape shop fucked around with this message at 08:55 on May 5, 2015

# ? May 5, 2015 08:46

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 14:35

QuarkJets: Sep 8, 2008

A Saucy Bratwurst posted:

Now i just need to figure out how to have the dictionary save across every instance of TemperatureData (and then do the same for the list) so it does the thing where it has every station that's been loaded so far.

I'm not sure why you have multiple instances of TemperatureData anyway, but okay. It seems like the assignment implies that you just need one TemperatureData instance

quote:

e: got it, but now theres nothing at all inside of __init__ but I don't think thats an issue is it?

If your class has any variables, it's considered good form to declare them in your __init__ even if you just assign None to them. But Python will let you get away with not doing that

# ? May 5, 2015 09:45

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »