Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Cingulate: Oct 23, 2012; by Fluffdaddy

Nippashish posted:

sklearn

Related to the question in the Math thread: if I use a kernel method, is the kernel "fit"? So for example, Kernel Ridge has a multi-target option. If I run it with target vector A vs. with target vectors A, B, C simultaneously, is the model learned for A different between the two runs?

And what about Random Forest, which also has a multi-target option: what are the pros and cons for running it on multiple targets simultaneously vs. in a row?

# ? Nov 26, 2016 14:40

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 19:28

Nippashish: Nov 2, 2005; Let me see you dance!

Cingulate posted:

Related to the question in the Math thread: if I use a kernel method, is the kernel "fit"? So for example, Kernel Ridge has a multi-target option. If I run it with target vector A vs. with target vectors A, B, C simultaneously, is the model learned for A different between the two runs?

And what about Random Forest, which also has a multi-target option: what are the pros and cons for running it on multiple targets simultaneously vs. in a row?

Are you doing classification or regression? Your earlier question was about SVC which is a classification model, but Kernel Ridge is a regression model. I may have directed you down a garden path by suggesting kernel logistic regression because in spite of its name it is actually a classification model, and because afaict it's not actually implemented in sklearn. Sorry if that was confusing.

Your question doesn't really make sense for classification because classification is about distinguishing between sets of mutually exclusive options. You can't fit a classification model on "just A" because classifiers answer a question like "is this thing A or B (or C or ...)?". Some models, like SVMs, can only make binary distinctions and in that case you can do different kinds of voting like what was going on with matching decision_function to predict for SVC in your previous post. That's about as close as you can get to "one at a time" for classification. For regression the story is different because "one at a time" makes sense, and whether or not you get the same answer with one at a time vs all at once depends on the model. For kernel ridge I believe you get the same thing both ways, but for random forests you would not.

My personal take on this is that it's best to use a model that can make the type of predictions you care about without an extra post-processing step. If you want to distinguish between many types of thing then it is better to use a classification model that can "natively" make multi-way decisions rather than to make many binary decisions and combine them. Similarly, if you care about measuring confidence of your predictions then it is better to use a model that encodes confidence in its decision surface than to fit an auxiliary calibration model after the fact.

A concrete suggestion is to pick one of the tree-based Classifier models from sklearn's ensemble package. RandomForestClassifier, GradientBoostingClassifier and ExtraTreesClassifier all support multi-class decisions and can produce confidences in their predictions natively. This approach also has the nice side effect that you don't need to think about kernels anymore.

# ? Nov 27, 2016 02:08

Cingulate: Oct 23, 2012; by Fluffdaddy

Nippashish posted:

Are you doing classification or regression? Your earlier question was about SVC which is a classification model, but Kernel Ridge is a regression model. I may have directed you down a garden path by suggesting kernel logistic regression because in spite of its name it is actually a classification model, and because afaict it's not actually implemented in sklearn. Sorry if that was confusing.

Your question doesn't really make sense for classification because classification is about distinguishing between sets of mutually exclusive options. You can't fit a classification model on "just A" because classifiers answer a question like "is this thing A or B (or C or ...)?". Some models, like SVMs, can only make binary distinctions and in that case you can do different kinds of voting like what was going on with matching decision_function to predict for SVC in your previous post. That's about as close as you can get to "one at a time" for classification. For regression the story is different because "one at a time" makes sense, and whether or not you get the same answer with one at a time vs all at once depends on the model. For kernel ridge I believe you get the same thing both ways, but for random forests you would not.

My personal take on this is that it's best to use a model that can make the type of predictions you care about without an extra post-processing step. If you want to distinguish between many types of thing then it is better to use a classification model that can "natively" make multi-way decisions rather than to make many binary decisions and combine them. Similarly, if you care about measuring confidence of your predictions then it is better to use a model that encodes confidence in its decision surface than to fit an auxiliary calibration model after the fact.

A concrete suggestion is to pick one of the tree-based Classifier models from sklearn's ensemble package. RandomForestClassifier, GradientBoostingClassifier and ExtraTreesClassifier all support multi-class decisions and can produce confidences in their predictions natively. This approach also has the nice side effect that you don't need to think about kernels anymore.

Eeeeeeeh I know the distinction between classification and regression. I use both a lot, though I usually stick with linear models for both tasks since they work best for my very particular usage cases. In this specific instance, I was using Kernel Ridge for regression, realized the RBF kernel works a lot better than the linear kernel and ordinary Ridge (and, for some reason, Random Forest Regression!?.. Though I actually have a decent idea why!), decided I should probably figure out what exactly these "Kernel" things are considering they work so nicely in this case, and phrased my question in the context of SVMs cause everything I found online was about SVMs and I assumed it'd be able to generalize back to the regression case. (I guess there's also SVR etc., but I'm sticking with Kernel Ridge due to the closed-form solvability, which is a huge benefit with the multitarget nature of my problems.)

Sorry for making you type all that text about classification vs. regression! But thanks for answering the question still.

I'm also looking at a few multilabel problems on occasion, but they're not high up on the list of priorities.

# ? Nov 27, 2016 03:34

huhu: Feb 24, 2006

After hours of frustration with a script I'm writing, I've come to realize I could have been using CSVs instead of Excel files. So, lesson learned, if you can deal with CSV files instead of Excel do it. If you're wondering why, Excel columns can start at 1 or "A" however everything in Python starts at 0.

# ? Nov 28, 2016 16:47

Dex: May 26, 2006; Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

huhu posted:

After hours of frustration with a script I'm writing, I've come to realize I could have been using CSVs instead of Excel files. So, lesson learned, if you can deal with CSV files instead of Excel do it. If you're wondering why, Excel columns can start at 1 or "A" however everything in Python starts at 0.

i use openpyxl for excel stuff

# ? Nov 28, 2016 19:13

Cingulate: Oct 23, 2012; by Fluffdaddy

Pandas can directly read xls, which is probably usually what you want anyways.

# ? Nov 28, 2016 21:03

vikingstrike: Sep 23, 2007; whats happening, captain

Yeah, just use pandas. CSV is usually more convenient though, IME, especially with larger files since you'll likely never want to open them ... in Excel. But we work in real office places (I'm assuming some of us at least) and you're sent what you're sent. I usually will take a raw file like that, clean/sanity check it, and then write an HDF file with the type info.

# ? Nov 29, 2016 05:35

ahmeni: May 1, 2005; It's one continuous form where hardware and software function in perfect unison, creating a new generation of iPhone that's better by any measure.; Grimey Drawer

Does anyone have a pattern for logging requests in Django Rest Framework that they like? The only real solution I've seen is a middleware that requires a mixin for each APIView, which isn't too bad but I'd prefer something a tad less fiddly.

# ? Nov 29, 2016 12:38

Space Kablooey: May 6, 2009

ahmeni posted:

Does anyone have a pattern for logging requests in Django Rest Framework that they like? The only real solution I've seen is a middleware that requires a mixin for each APIView, which isn't too bad but I'd prefer something a tad less fiddly.

Isn't this the kind of thing that it's better if you do in your server instead of the application?

# ? Nov 29, 2016 13:28

ahmeni: May 1, 2005; It's one continuous form where hardware and software function in perfect unison, creating a new generation of iPhone that's better by any measure.; Grimey Drawer

HardDiskD posted:

Isn't this the kind of thing that it's better if you do in your server instead of the application?

Generally yes and I already log at the nginx/uwsgi level for regular http stuff. We have a few cases where programmatic control of logging at the incoming request level would be handy to have. Looks like mixins are the way to go though.

# ? Nov 30, 2016 08:49

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

If I want to import say just sqrt from numpy I could do

code:


from numpy import sqrt

But what do I do if I want to import just sqrt but keep the same numpy namespace?

code:


from numpy import sqrt as numpy_sqrt
from numpy import sqrt as numpy.sqrt

The first one works but I'd rather have it called as numpy.sqrt still. The second one doesn't seem to work.

# ? Nov 30, 2016 11:31

Cingulate: Oct 23, 2012; by Fluffdaddy

Actually, we

code:

import numpy as np

# ? Nov 30, 2016 12:53

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Boris Galerkin posted:

If I want to import say just sqrt from numpy I could do
code:
from numpy import sqrt
But what do I do if I want to import just sqrt but keep the same numpy namespace?
code:
from numpy import sqrt as numpy_sqrt
from numpy import sqrt as numpy.sqrt
The first one works but I'd rather have it called as numpy.sqrt still. The second one doesn't seem to work.

I don't think there's a non convoluted way to do that. Keep in mind that even when you do "from numpy import sqrt" instead of "import numpy", python still needs to parse the whole numpy module to bind that function. So in my opinion you should just do "import numpy" anyway.

# ? Nov 30, 2016 17:45

accipter: Sep 12, 2003

Symbolic Butt posted:

I don't think there's a non convoluted way to do that. Keep in mind that even when you do "from numpy import sqrt" instead of "import numpy", python still needs to parse the whole numpy module to bind that function. So in my opinion you should just do "import numpy" anyway.

Or stick with the convention of "import numpy as np".

# ? Nov 30, 2016 19:23

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

accipter posted:

Or stick with the convention of "import numpy as np".

Sure, I agree with the convention, but renaming the module namespace is not all that relevant to what Boris asked.

# ? Nov 30, 2016 19:52

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Is numpy vs np a religious thing like vim vs emacs?

e: I guess if you're forced to 79 characters (lololol what year is this) then that makes sense yeah.

# ? Nov 30, 2016 22:24

Nippashish: Nov 2, 2005; Let me see you dance!

Its a convention that more or less everyone follows, like using snake_case instead of camelCase for variables or naming classes with PascalCase instead of something else. I've never encountered anyone with strong opinions about it though.

# ? Nov 30, 2016 22:30

Cingulate: Oct 23, 2012; by Fluffdaddy

Boris Galerkin posted:

Is numpy vs np a religious thing like vim vs emacs?

e: I guess if you're forced to 79 characters (lololol what year is this) then that makes sense yeah.

The year is 2016 and we follow pep8.

# ? Nov 30, 2016 22:52

Ghost of Reagan Past: Oct 7, 2003; rock and roll fun

Cingulate posted:

The year is 2016 and we follow pep8.

praise be to pep8

# ? Dec 1, 2016 01:51

IAmKale: Jun 7, 2007; やらないか; Fun Shoe

Ghost of Reagan Past posted:

praise be to pep8

Embrace the darkness, increase your line length to 99 you heathens :argh:

# ? Dec 1, 2016 02:18

Dominoes: Sep 20, 2007

Nippashish posted:

Its a convention that more or less everyone follows, like using snake_case instead of camelCase for variables or naming classes with PascalCase instead of something else. I've never encountered anyone with strong opinions about it though.

Technique I use: import numpy as np standard for the most part. If I'm doing anything with messy math, I'll import the funcs I need separately, ie from numpy import sqrt, sin, cos, arctan2, pi, e etc to make equations easier to read.

# ? Dec 1, 2016 05:43

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

IAmKale posted:

Embrace the darkness, increase your line length to 99 you heathens

the best way to stick to 99 characters is to pretend your limit is 79 imo

# ? Dec 1, 2016 12:57

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

Character limits are way out of date. The only rule that should be applied is "don't take the piss".

# ? Dec 1, 2016 13:43

Dex: May 26, 2006; Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

[flake8]
max_complexity = 10
max_line_length = 9999

# ? Dec 1, 2016 14:28

ufarn: May 30, 2009

Can anyone spot what�s preventing me from installing PyAudio?

code:

$ pip install -U PyAudio
Collecting PyAudio
  Using cached PyAudio-0.2.9.tar.gz
Installing collected packages: PyAudio
  Running setup.py install for PyAudio ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, \
        tokenize;__file__='/private/var/folders/8f/hm_q__wj6sl7pv68v6fkw7080000gn/T/pip-build-z4WD4x/PyAudio/setup.py';f=getattr( \
        tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" \
        install --record /var/folders/8f/hm_q__wj6sl7pv68v6fkw7080000gn/T/pip-mCnzbJ-record/install-record.txt \
        --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.12-intel-2.7
    copying src/pyaudio.py -> build/lib.macosx-10.12-intel-2.7
    running build_ext
    building '_portaudio' extension
    creating build/temp.macosx-10.12-intel-2.7
    creating build/temp.macosx-10.12-intel-2.7/src
    cc -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -I/usr/local/opt/openssl/include \
    -I/usr/local/opt/openssl/include -pipe -arch -DMACOSX=1 \
    -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c \
    src/_portaudiomodule.c -o build/temp.macosx-10.12-intel-2.7/src/_portaudiomodule.o
    clang: error: invalid arch name '-arch -DMACOSX=1'
    error: command 'cc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__=\
'/private/var/folders/8f/hm_q__wj6sl7pv68v6fkw7080000gn/T/pip-build-z4WD4x/PyAudio/setup.py';f=getattr(\
tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" \
install --record /var/folders/8f/hm_q__wj6sl7pv68v6fkw7080000gn/T/pip-mCnzbJ-record/install-record.txt \
--single-version-externally-managed --compile" \
failed with error code 1 in /private/var/folders/8f/hm_q__wj6sl7pv68v6fkw7080000gn/T/pip-build-z4WD4x/PyAudio/

Had to break it up with \; here�s the unedited version: http://dpaste.com/3VKQF14.

ufarn fucked around with this message at 15:31 on Dec 1, 2016

# ? Dec 1, 2016 15:27

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Did you brew install portaudio before that?

# ? Dec 1, 2016 16:36

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

code:

    clang: error: invalid arch name '-arch -DMACOSX=1'

Somehow it can't detect your system architecture. I have no idea how this works and I really don't want to dig into setuptools right now. Try ARCHFLAGS='-arch x86_64' pip install -U PyAudio

# ? Dec 1, 2016 18:29

ufarn: May 30, 2009

Symbolic Butt posted:

Did you brew install portaudio before that?

Nope. It doesn't show up in `brew list` either.

Suspicious Dish posted:

code:
    clang: error: invalid arch name '-arch -DMACOSX=1'
Somehow it can't detect your system architecture. I have no idea how this works and I really don't want to dig into setuptools right now. Try ARCHFLAGS='-arch x86_64' pip install -U PyAudio

Same error.

# ? Dec 1, 2016 20:09

Cingulate: Oct 23, 2012; by Fluffdaddy

In scikit-learn, how can I best predict ratios/percentages with a linear model? I guess I could use trees or something, but is there a linear model that makes continuous predictions on the range (0, 1) available in sklearn?

I assume this is a stupid question and the answer is I'm not understanding how to properly use Logistic Regression. Or should I just use the inverse hyperbolic on my outcomes and go really linear?

# ? Dec 2, 2016 01:02

lifg: Dec 4, 2000; <this tag left blank>; Muldoon

Does anyone have experience or recommendations on building Python software with virtualenv and Docker?

I've used them both for toy applications, but not together. They seem to solve the same problem, just at different levels, so I don't *think* there's a benefit to using virtualenv if I'm also using Docker.

I've never used virtualization (of any sort) for software development, so this is all new to me.

# ? Dec 2, 2016 19:55

Space Kablooey: May 6, 2009

lifg posted:

Does anyone have experience or recommendations on building Python software with virtualenv and Docker?

I've used them both for toy applications, but not together. They seem to solve the same problem, just at different levels, so I don't *think* there's a benefit to using virtualenv if I'm also using Docker.

I've never used virtualization (of any sort) for software development, so this is all new to me.

I'm also quite new to Docker but in my experience, yeah, you really don't exactly need to ensure that the container's Python environment is clear, after all that's what the Docker container is for.

I'd love to hear from someone that has more experience with Docker, because I still can't wrap my head around on how to do some things, most importantly on how to update code and few other stuff.

# ? Dec 2, 2016 20:20

BigRedDot: Mar 6, 2008

ufarn posted:

Can anyone spot what�s preventing me from installing PyAudio?

Dunno what's wrong here, but if Anaconda/conda are an option, there is a pre-built conda package for PyAudio in the default repo, I use it on OS X and it works great.

# ? Dec 2, 2016 21:07

LochNessMonster: Feb 3, 2005; I need about three fitty

HardDiskD posted:

I'm also quite new to Docker but in my experience, yeah, you really don't exactly need to ensure that the container's Python environment is clear, after all that's what the Docker container is for.

I'd love to hear from someone that has more experience with Docker, because I still can't wrap my head around on how to do some things, most importantly on how to update code and few other stuff.

I've only used venv once so I probably shouldn't say anything about it. But isn't the whole point of venv to have a virtualized environment for 1 specific version of python and specific packages.

That's what you would pack into 1 container. If you have the need to run different applications with different python/package versions you'd normally do that in another container.

I'm really new to programming in general so I might be completely off here. The idea behind docker is to create microprocesses and put eacht functionality/program/service in it's own container and the containers talk to eachother.

# ? Dec 2, 2016 21:49

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

No need to use a virtual env if you're running your code in a container that I can think of. Just install your packages and python version "system wide" in the Docker container.

# ? Dec 2, 2016 21:53

Smugworth: Apr 18, 2003

I'm no expert in the least but I helped write Python code for starting Docker containers for my internship. I used venvs for my coding environment, but our Python code spawned containers running Java applications. It was pretty handy to work with, even though I didn't write the Docker providers. Docker-py is handy.

# ? Dec 3, 2016 07:35

Dex: May 26, 2006; Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

lifg posted:

Does anyone have experience or recommendations on building Python software with virtualenv and Docker?

I've used them both for toy applications, but not together. They seem to solve the same problem, just at different levels, so I don't *think* there's a benefit to using virtualenv if I'm also using Docker.

I've never used virtualization (of any sort) for software development, so this is all new to me.

what are you using for your builds? we use a swarm and yet another docker plugin to run our builds - each build spawns a brand new container so there's no need for virtualenv. if you're intending to use docker compose to stand up a bunch of build slaves that'll be persisted for more than a single build, you'd want virtualenv

# ? Dec 3, 2016 19:11

lifg: Dec 4, 2000; <this tag left blank>; Muldoon

Dex posted:

what are you using for your builds? we use a swarm and yet another docker plugin to run our builds - each build spawns a brand new container so there's no need for virtualenv. if you're intending to use docker compose to stand up a bunch of build slaves that'll be persisted for more than a single build, you'd want virtualenv

The answer is mostly "that's a good question and I wish I had a good answer."

The basic setup uses OpenShift. I pick one of a few generic images built for my language of choice, then hook it into a Git repository with my code. Then every time I push the master branch the container is automatically rebuilt.

I don't actually use the docker command for any of the above, but I'm guessing I'll need to for development.

# ? Dec 3, 2016 20:00

Dex: May 26, 2006; Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

i haven't used openshift but i see you have a couple of options there - custom dockerfiles, hooking in with jenkins pipelines(which i've worked with _a lot_), and more. so it's really a question of how you want to work. if you just want to code in your ide and run your tests etc locally, and ignore the final artifact that pops out of your build system, then you don't need docker locally. your build process could just be running a custom dockerfile that installs your deps and runs your unit tests as part of every commit, but only pushes the image on merges to master. so something like

code:

FROM my-base-python-image:whatever

RUN mkdir -p /opt/myapp
ADD . /opt/myapp/
WORKDIR /opt/myapp/
RUN pip install -r requirements.txt && python -m unittest discover tests/ && flake8 && rm -rf tests/
CMD ["howeveryourunyourapp"]

so if you wanted a local image, you'd just run "docker build -t mydevimage ." in your app directory and it would execute the above, which you could then run with "docker run mydevimage". or you could just leave your build systems do the docker bits that they need for deployment, and maintain your local dev environment however you want.

Dex fucked around with this message at 22:39 on Dec 3, 2016

# ? Dec 3, 2016 22:13

Cockmaster: Feb 24, 2002

What are some good resources for learning GUI programming in Python? I'll be writing some interface software for a robot - nothing too fancy, just standard off-the-shelf GUI objects. Plus I'd need the code to run on either a Windows 10 system or a Raspberry Pi, so I would assume I'd want to use something like Tkinter or wxPython.

# ? Dec 5, 2016 16:55

Adbot: ADBOT LOVES YOU

# ? May 8, 2024 19:28

Tigren: Oct 3, 2003

Cockmaster posted:

What are some good resources for learning GUI programming in Python? I'll be writing some interface software for a robot - nothing too fancy, just standard off-the-shelf GUI objects. Plus I'd need the code to run on either a Windows 10 system or a Raspberry Pi, so I would assume I'd want to use something like Tkinter or wxPython.

Any hangups making a small web app and then you can design the GUI using HTML5/CSS/JS?

# ? Dec 5, 2016 18:01

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »