Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

mbt: Aug 13, 2012

I'm trying to use python to make my data analysis more automated but I can't seem to find a way to use its csv module to perform operations between columns.

I've got a column called "current" and a column called "voltage". I want this program to take each line of voltage and divide it by each line of current and put the resistance in a third column.

I've tried both reader and dictreader methods, dictreader being easier to convert both into floats, but I can't seem to do the line by line division. I think csv might work like making an array of arrays, at least that's what I think after voltage[2] returns "l" (in non-dictreader). Anyone have experience with csv or can point me to a solution? I feel like I've looked nearly everywhere and read the documentation a hundred times already.

# ? Nov 17, 2013 04:32

Adbot: ADBOT LOVES YOU

# ? May 20, 2024 09:09

Nippashish: Nov 2, 2005; Let me see you dance!

Mortimer posted:

I'm trying to use python to make my data analysis more automated but I can't seem to find a way to use its csv module to perform operations between columns.

I've got a column called "current" and a column called "voltage". I want this program to take each line of voltage and divide it by each line of current and put the resistance in a third column.

I've tried both reader and dictreader methods, dictreader being easier to convert both into floats, but I can't seem to do the line by line division. I think csv might work like making an array of arrays, at least that's what I think after voltage[2] returns "l" (in non-dictreader). Anyone have experience with csv or can point me to a solution? I feel like I've looked nearly everywhere and read the documentation a hundred times already.

You should use pandas for dealing with tables of data in python.

# ? Nov 17, 2013 05:09

My Rhythmic Crotch: Jan 13, 2011

Not going to win any beauty pageants, but it works.

Python code:

#!/usr/bin/env python

import csv

output = []
with open('sample.csv', 'r') as f_in:
	reader = csv.reader(f_in)
	for row in reader:
		current = row[0]
		voltage = row[1]
		resistance = float(voltage) / float(current)
		output.append((current,voltage,resistance))
with open ('output.csv', 'w') as f_out:
	writer = csv.writer(f_out)
	for row in output:
		writer.writerow(row)

# ? Nov 17, 2013 05:12

Dren: Jan 5, 2001; Pillbug

the, have you heard of lists? and joining them?

e.g.

Python code:

l = ['some', 'crap']

print '\n'.join(l)

# ? Nov 17, 2013 05:46

Dominoes: Sep 20, 2007

Is there a trick to getting PyCharm professional to recognize imports properly for the purposes of warnings?

'Red warning: No module named mymodule' for each .py file I import from my own project.

Yellow cannot find reference warning for many other things, such as 'Cannot find reference 'pyqtSignal' in 'QtCore.py' Most (all?) Qt objects have. I get the same message for other things, ie

Python code:

avg_time = sum(times_owned, dt.timedelta()) / len(times_owned)
days_fractional = (avg_time.days + avg_time.seconds/86400)

gives 'Unresolved attribute reference 'days' for class 'int'. I guess PyCharm's assuming avg_time is an int (it's a datetime.timedelta object) because I did arithmetic on it?

Django code in HTML's also flagged.

The Pycharm warnings seem great in principle, but sorting out the chaff's a pain.

Dominoes fucked around with this message at 18:22 on Nov 17, 2013

# ? Nov 17, 2013 17:30

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

Is there a trick to getting PyCharm professional to recognize imports properly for the purposes of warnings?

'Red warning: No module named mymodule' for each .py file I import from my own project.

Yellow cannot find reference warning for many other things, such as 'Cannot find reference 'pyqtSignal' in 'QtCore.py' Most (all?) Qt objects have. I get the same message for other things, ie
Python code:
avg_time = sum(times_owned, dt.timedelta()) / len(times_owned)
days_fractional = (avg_time.days + avg_time.seconds/86400)
gives 'Unresolved attribute reference 'days' for class 'int'. I guess PyCharm's assuming avg_time is an int (it's a datetime.timedelta object) because I did arithmetic on it?

Django code in HTML's also flagged.

The Pycharm warnings seem great in principle, but sorting out the chaff's a pain.

It pretty much Just Works. Sounds like you've got the project configured wrong. Are you using a virtualenv? (you better be!) Have you configured the project interpreter so PyCharm knows where to look for your modules?

# ? Nov 17, 2013 18:42

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Double post because I'm awesome.

The new OP is coming along. Posting it once more for anything further people would like in it. Don't forget you can comment right on the google doc.

I started writing out examples for a bunch of the libraries and stuff, but man I'm feeling pretty lazy and that will end up being a huge job.

Other things I know need some work:

Some descriptions need fleshed out.
Some more beginner-type information like...I mention some unit test libraries, but some info on why you should unit test might be a good idea.

# ? Nov 17, 2013 18:47

Dominoes: Sep 20, 2007

Thermopyle posted:

It pretty much Just Works. Sounds like you've got the project configured wrong. Are you using a virtualenv? (you better be!) Have you configured the project interpreter so PyCharm knows where to look for your modules?

I haven't figured out the project system; I'm just using it edit python files.

# ? Nov 17, 2013 19:03

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Dominoes posted:

I haven't figured out the project system; I'm just using it edit python files.

What does your file structure look like? What do your import statements look like?

# ? Nov 17, 2013 19:10

Dominoes: Sep 20, 2007

BeefofAges posted:

What does your file structure look like? What do your import statements look like?

-code (I set this as PyCharm's project directory)
----myprogram
--------script1.py
--------script2.py

Code from script1.py

Python code:

import script2

Dominoes fucked around with this message at 19:38 on Nov 17, 2013

# ? Nov 17, 2013 19:36

Bloodborne: Sep 24, 2008

Thermopyle posted:

Double post because I'm awesome.

The new OP is coming along. Posting it once more for anything further people would like in it. Don't forget you can comment right on the google doc.

I started writing out examples for a bunch of the libraries and stuff, but man I'm feeling pretty lazy and that will end up being a huge job.

Other things I know need some work:

Some descriptions need fleshed out.
Some more beginner-type information like...I mention some unit test libraries, but some info on why you should unit test might be a good idea.

Thanks for doing this, as a new to programming and new to Python goon. A lot of us newbees I'm sure realize it's a pretty big undertaking fleshing stuff out. Python is fun and I'm having fun taking it slowly and learning stuff a little at a time as a side.

# ? Nov 17, 2013 20:32

Dominoes: Sep 20, 2007

Fun fact: Web forms, such as when using django, use the characters '\r\n' to signify a line break. I spent hours troubleshooting a mystery issue parsing a text file with regexes from a django app, until stumbling across the answer, hidden in the bowels of stack overflow. The regex \s can find this line break, but I had to replace '\r\n' explicitly to format it properly.

# ? Nov 17, 2013 20:40

Lurchington: Jan 2, 2003; Forums Dragoon

Dominoes posted:

-code (I set this as PyCharm's project directory)
----myprogram
--------script1.py
--------script2.py

Code from script1.py
Python code:
import script2

if you mark "myprogram" as "Source Root" by right clicking the folder in the project explorer that will help.

It's ambiguous if "code" is actually the directory for this project, or the overall project directory, which could possibly explain why pycharm needs to be taught about this (especially since you're probably using the system python instead of a virtual environment)

# ? Nov 17, 2013 21:40

Dominoes: Sep 20, 2007

Lurchington posted:

if you mark "myprogram" as "Source Root" by right clicking the folder in the project explorer that will help.

It's ambiguous if "code" is actually the directory for this project, or the overall project directory, which could possibly explain why pycharm needs to be taught about this (especially since you're probably using the system python instead of a virtual environment)

Cool, that worked for the import issue. I'm using the code directory, since I'm working on multiple projects simultaneously. I don't understand, nor at this point want to learn PyCharm's project system. I'm using it for its on-the-spot error-catching and suggestions.

# ? Nov 17, 2013 21:48

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

Cool, that worked for the import issue. I'm using the code directory, since I'm working on multiple projects simultaneously. I don't understand, nor at this point want to learn PyCharm's project system. I'm using it for its on-the-spot error-catching and suggestions.

FYI, You literally can't use PyCharm without using projects.

# ? Nov 17, 2013 21:51

Pollyanna: Mar 5, 2005; Milk's on them.

The project system isn't too hard. It's basically just organizing all your code together, like a folder, and making sure everything's using the same interpreter, environment, etc.

# ? Nov 18, 2013 15:43

the: Jul 18, 2004; by Cowcaster

For anyone who wants to try, here's the programming test I was given for a Junior Python position at a company:

quote:

PML Project
-A PML document is a standard HTML document with one additional feature. Any text between the starting <pml> tag and the ending </pml> tag is interpreted as Python source code.
-There can be multiple PML blocks in a PML file.
-PML blocks will never nest.
-The standard HTML should pass through the parser untouched.
-The code within the PML tags should be executed with the python interpreter.
-A technique should be implemented to write data to the output stream from within the PML. In other words the python code should be able to define the output that will replace the PML.
-Variables and functions declared in one PML block should be available in subsequent PML blocks.
-PML should be able to handle indentation dependent upon the first non-whitespace line of python code.

Example Input:
code:
<html>
<h1>Hello There</h1>
 
<p>
This is an example of a pml file
</p>
 
<pml>
    def f():
        return "<h2>Good Bye</h2>"
 
    pml = f()
</pml>
 
<html>
Example Output:
code:
<html>
<h1>Hello There</h1>
 
<p>
This is an example of a pml file
</p>
 
<h2>Good Bye</h2>
 
<html>

# ? Nov 18, 2013 18:27

Lurchington: Jan 2, 2003; Forums Dragoon

I'm curious, were there any warnings not to use exec/eval?

# ? Nov 18, 2013 18:33

the: Jul 18, 2004; by Cowcaster

Lurchington posted:

I'm curious, were there any warnings not to use exec/eval?

No, that's the entirety of the instructions aside from "email it to us at this date blah blah."

# ? Nov 18, 2013 18:35

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I typed up my quick solution, but instead I want to know what your solution was.

Also, as I've never been paid to program, I wonder how typical that type of question is. Because that makes me feel a lot better about my prospects should I ever go looking for a Python job.

# ? Nov 18, 2013 18:56

SelfOM: Jun 15, 2010

I find myself using this pattern a lot for testing for numpy arrays as optional arguments. Is this the best way to do this?

Python code:

import numpy as np

def test_func(x, y=None):
    if getattr(y, 'shape', False):
         print('Array passed for y')
    else:
         print('No array passed for y')

# ? Nov 18, 2013 19:56

Pollyanna: Mar 5, 2005; Milk's on them.

the posted:

PML

I can't be the only one who thinks this is a horrible idea, right? Aren't you not supposed to execute arbitrary Python in a webpage or something?

Fake edit: Apparently that's sort of what mod_python does, but I'm still weirded out by that test. I can foresee nothing but tears coming from this.

# ? Nov 18, 2013 20:22

the: Jul 18, 2004; by Cowcaster

Thermopyle posted:

I typed up my quick solution, but instead I want to know what your solution was.

Also, as I've never been paid to program, I wonder how typical that type of question is. Because that makes me feel a lot better about my prospects should I ever go looking for a Python job.

Python code:

import os
from textwrap import dedent

os.chdir('C:\Users\Matthew')

p = True
e = False

with open('pml.html', 'r') as f:
    for line in f:
        if '<pml>' in line:
            p = False
            e = True
        if '</pml>' in line:
            p = True
            e = False
        if p == True and e == False:
            if '</pml>' in line:
                execfile('tmp.py')
            else:
                print line
        if p == False and e == True:
            if '<pml>' in line:
                pass
            else:
                with open('tmp.py', 'a') as t:
                    t.write(line)
                    
os.remove('tmp.py')

The only issue is that my code doesn't handle the initial indent after the <pml> tag in their example code, which seems totally pointless. If I remove the initial indent the code runs fine and it doesn't affect the look much at all.

# ? Nov 18, 2013 21:15

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Pollyanna posted:

I can't be the only one who thinks this is a horrible idea, right? Aren't you not supposed to execute arbitrary Python in a webpage or something?

Fake edit: Apparently that's sort of what mod_python does, but I'm still weirded out by that test. I can foresee nothing but tears coming from this.

Browsers don't support this so-called pml tag. In fact, nothing supports it.

Now, think about how this affects your questions.

# ? Nov 18, 2013 21:18

My Rhythmic Crotch: Jan 13, 2011

the posted:

For anyone who wants to try, here's the programming test I was given for a Junior Python position at a company:

"Do you care about security? No? Great! Come work for PML, Inc!"

# ? Nov 18, 2013 21:27

Haystack: Jan 23, 2005

Pollyanna posted:

I can't be the only one who thinks this is a horrible idea, right? Aren't you not supposed to execute arbitrary Python in a webpage or something?

Fake edit: Apparently that's sort of what mod_python does, but I'm still weirded out by that test. I can foresee nothing but tears coming from this.

Technically, this PML thing is just a template DSL like Jinja, Chameleon, or Mako. Mako in particular lets you slap arbitrary python blocks into your templates, although you're only supposed to do that for complicated presentational calculations if at all.

That said, that PML thing would be pretty horrifying if it were anything other than a test question.

# ? Nov 18, 2013 21:33

Nippashish: Nov 2, 2005; Let me see you dance!

Pollyanna posted:

Fake edit: Apparently that's sort of what mod_python does, but I'm still weirded out by that test. I can foresee nothing but tears coming from this.

I don't really see how pml is inherently less secure than having your browser run whatever shifty javascript code a webpage pulls in.

# ? Nov 18, 2013 21:34

Plorkyeran: Mar 22, 2007; To Escape The Shackles Of The Old Forums, We Must Reject The Tribal Negativity He Endorsed

Nippashish posted:

Mod python is for running python server side, presumably this pml thing is intended to run it client side

What makes you think that?

There's nothing inherently insecure or even remotely unusual about the idea of PML.

# ? Nov 18, 2013 21:38

evensevenone: May 12, 2001; Glass is a solid.

It's obviously test to see if you can write (or use) a parser and do a trivial task with the result, not of best practices in system architecture.

# ? Nov 18, 2013 21:48

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

the posted:

The only issue is that my code doesn't handle the initial indent after the <pml> tag in their example code, which seems totally pointless. If I remove the initial indent the code runs fine and it doesn't affect the look much at all.

Your code won't handle multiple PML blocks according to the spec.

code:

foo
<pml>
print "hello"
</pml>
bar
<pml>
print "hi"
</pml>
baz

will output:

code:

foo
hello
bar
hello
hi
baz

# ? Nov 18, 2013 21:52

Lurchington: Jan 2, 2003; Forums Dragoon

Pollyanna posted:

I can't be the only one who thinks this is a horrible idea, right? Aren't you not supposed to execute arbitrary Python in a webpage or something?

Fake edit: Apparently that's sort of what mod_python does, but I'm still weirded out by that test. I can foresee nothing but tears coming from this.

this is one of the main criticisms against the cheetah templating engine too

the fact that you're appending to the same file and executing it multiple times, with exec file means that you can end up with weirdness on the offchance they write out pml that has side effects along with it.

for example, in the intentionally bad code:

code:

<pml>
import uuid
def buttes():
    _temp_var = 'uniquevar%s' % uuid.uuid4()
    globals()[_temp_var.replace('-', '_')] = 'butts'

buttes()
</pml>
<pml>
</pml>
<pml>
</pml>
<pml>
</pml>
<pml>
def butz()
    return len([unique_key for unique_key in globals() if unique_key.startswith('uniquevar')]

pml = butz()
</pml>

I think the posted snippet will return 5, but the solution probably wanted to return 1
I guess what I'm saying is that execfile will put things in your modules globals(), and the fact that you're re-executing code via append seems bad.

also, my preferred way to mess with this code:

code:

<pml>
import itertools
pml = list(itertools.cycle('\n'.join('butts'))
</pml>

# ? Nov 18, 2013 21:53

the: Jul 18, 2004; by Cowcaster

Oh, well. Hopefully they see past that. They said they intentionally gave me something "I wasn't familiar with," knowing that I come from a background of numerical computation, etc.

# ? Nov 18, 2013 22:47

mbt: Aug 13, 2012

My Rhythmic Crotch posted:

You are glorious, thank you very much. Worked perfectly.

And I'm definitely going to learn pandas.

# ? Nov 18, 2013 23:39

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

There's nothing more insecure about this PML thingy than with, say Django's templating system.

Browsers can't do anything with it, so it would only be used on the server side.

The only thing that can do anything with it is whatever server side tools the company would use to generate HTML with it, and that's exactly the case with any other templating thing.

Of course, it's just a test question, but this was brought up by this:

Pollyanna posted:

I can't be the only one who thinks this is a horrible idea, right? Aren't you not supposed to execute arbitrary Python in a webpage or something?

Fake edit: Apparently that's sort of what mod_python does, but I'm still weirded out by that test. I can foresee nothing but tears coming from this.

In other words, PML would be exactly as secure as any other dynamically generated website on the internet.

# ? Nov 18, 2013 23:54

Haystack: Jan 23, 2005

Thermopyle posted:

There's nothing more insecure about this PML thingy than with, say Django's templating system.

Well, except that most template engines offer automatic HTML escaping.

# ? Nov 19, 2013 00:03

Pollyanna: Mar 5, 2005; Milk's on them.

Thermopyle posted:

Browsers don't support this so-called pml tag. In fact, nothing supports it.

Now, think about how this affects your questions.

Which is why this is so :psyduck:

to me. What is even the point of this test? Why?

# ? Nov 19, 2013 00:10

fritz: Jul 26, 2003

Pollyanna posted:

Which is why this is so to me. What is even the point of this test? Why?

To see if someone can follow basic instructions and has a minimum level of competence.

# ? Nov 19, 2013 00:12

fritz: Jul 26, 2003

"What? Why would anybody want a tool that prints Fizz, Buzz, or Fizzbuzz?!?!?"

# ? Nov 19, 2013 00:13

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Pollyanna posted:

Which is why this is so to me. What is even the point of this test? Why?

Well, that's not what you questioned originally. Your original objection was that you're not supposed to execute Python in a web page....and that's not what this does.

fritz covered as to why this would be in an interview.

# ? Nov 19, 2013 00:28

Adbot: ADBOT LOVES YOU

# ? May 20, 2024 09:09

My Rhythmic Crotch: Jan 13, 2011

Thermopyle posted:

There's nothing more insecure about this PML thingy than with, say Django's templating system.

Browsers can't do anything with it, so it would only be used on the server side.

The only thing that can do anything with it is whatever server side tools the company would use to generate HTML with it, and that's exactly the case with any other templating thing.

Of course, it's just a test question, but this was brought up by this:

In other words, PML would be exactly as secure as any other dynamically generated website on the internet.

I'm just going to quote this page, replacing PHP with PML.

Someone way smarter than me posted:

Imagine you have a file upload form that dumps files into some public directory. To make sure nobody uploads PML files, you just check that they don�t have a .pml extension. All an attacker has to do is upload a file named foo.pml.txt; your uploader won�t see a problem, but Apache will recognize it as PML, and it will happily execute.

Granted, it's not exactly the same, but you get the idea. Also don't take what he says as gospel, because this is what his desk looks like.

# ? Nov 19, 2013 01:53

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »