Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
OnceIWasAnOstrich
Jul 22, 2006

Whichever method you are using to download the csv, save it into a string and then use cStringIO to turn that string into a file handle which you feed to csv.reader().

Adbot
ADBOT LOVES YOU

evilentity
Jun 25, 2010

Dominoes posted:

Another CSV / memory vs drive write question. If I want to download a CSV from a url, is there a way I can put the file information directly into a variable in memory? I've tried many combinations using list(), read(), csv.reader() etc, but can't find a working solution that skips writing the csv to disk, and reading it.

Code like this works:
Python code:
    def function():
        u = urlopen('http://somethingawful.com/COBOL.csv')
        with open('drive_file.csv', 'wb') as f:
            f.write(u.read())
        
        with open('drive_file.csv', 'r') as f:
            return list(csv.reader(f))
For example, this seems like it should do what I'm asking, but doesn't.

Python code:
    def function():
        u = urlopen('http://somethingawful.com/COBOL.csv')
        return list(csv.reader(u.read()))

It doesnt work because reader expects file handle not string.

Maybe you can use StringIO to makes this easy like this:
Python code:
import StringIO    
def function():
    u = urlopen('http://somethingawful.com/COBOL.csv').read()
    return list(csv.reader(StringIO.StringIO(u)))
Ive never used this module so i dunno. Maybe theres better way.

Dren
Jan 5, 2001

Pillbug

Dominoes posted:

For example, this seems like it should do what I'm asking, but doesn't.

Python code:
    def function():
        u = urlopen('http://somethingawful.com/COBOL.csv')
        return list(csv.reader(u.read()))

Did you try:
Python code:
    def function():
        u = urlopen('http://somethingawful.com/COBOL.csv')
        return list(csv.reader(u))

Experto Crede
Aug 19, 2008

Keep on Truckin'
I'm trying to use subprocess.call to run a wget command which uses a variable as part of it.

The code is:
Python code:
content = subprocess.call(['wget', 'website/check.php?name=', name])
with name being a variable based on a raw_input. However, it seems that this puts a space between the url and name, even if there's no space in the code. All this page has on it is either a 1 or a 0 dependent on whether the value of name is something present on the remote system, and I want that to be the value of content. Maybe there's a better way to do it?

Either way, any advice?

NOTinuyasha
Oct 17, 2006

 
The Great Twist
Python code:
content = subprocess.call(["wget","website/check.php?name={0}".format(name)])
Is the straight answer. But really...

Python code:
import requests

content = requests.get("http://google.com").text
Edit: Below is a better answer.

NOTinuyasha fucked around with this message at 18:40 on Apr 6, 2013

Lil PP
Apr 1, 2010

hi my name is billy smith. whatchuwan? the police hav orders 4 u 2 cum ovr 2 mi house if ur a hot guy lookin 2 smoke some weed nkd nd j/o 2 the best of 90's nick. do u have what it takes? i guess we'll FIGURE IT OUT (must not b shy about being slimd or listenin 2 dubstep/flashbulb, pm if interested)

NOTinuyasha posted:

Is the straight answer. But really...

Python code:
import requests

content = requests.get("http://google.com").text

And if you're only able to use built in libraries:

Python code:
# At the top of your file
import urllib

# wherever, assuming that page only returns 0 and 1
result = bool(urllib.urlopen('website.com/check.php?name={0}'.format(name)).read())

if result:
  print 'remote system is running whatever'
else:
  print 'bzzzzzt'

Dren
Jan 5, 2001

Pillbug

BigRedDot posted:

Hi guys, I just wanted to throw out that the videos of tutorials and talks from our latest PyData conference are up on vimeo now:

http://vimeo.com/pydata/videos

I think the keynotes are still missing, but everything else (about 30 videos, including my talk! but which one is it?) is available There are great talks and tutorials on tools: numpy, matplotlib, pandas, scikit-learn, as well as a variety of specific data analysis examples involving python. Hope you might find something interesting or useful!

Did you do the luigi talk? How does hadoop attach a module thing work? Does hadoop have a provision for it or are you sending bytecode somehow? (I am a hadoop noob).

BigRedDot
Mar 6, 2008

Dren posted:

Did you do the luigi talk? How does hadoop attach a module thing work? Does hadoop have a provision for it or are you sending bytecode somehow? (I am a hadoop noob).

No, sorry I shouldn't have been so coy. I gave the Numpy tutorial.

maniacripper
May 3, 2009
STANNIS BURNS SHIREEN
HIZDAR IS THE HARPY
JON GETS STABBED TO DEATH
DANY FLIES OFF ON DROGON
So I'm starting to learn Python (2.7.3) and I have absolutely zero programming/scripting experience.

Should I bother with ver2 at all? Should I just start learning ver3?


Either way I'm doing some exercises and I can't get it to math properly.

It's a simple math operation where I want to print the average of 2 values.
The program looks like this:

x = 3
y = 4
average = (x + y) / 2
print(average)
3 <----- this poo poo right here is wrong wrong wrong!

What am i doing wrong?

evensevenone
May 12, 2001
Glass is a solid.
It's guessing that you want to store 3 and 4 as ints, and rounding down the result so it's also an int. Try 3.0 and 4.0.

maniacripper
May 3, 2009
STANNIS BURNS SHIREEN
HIZDAR IS THE HARPY
JON GETS STABBED TO DEATH
DANY FLIES OFF ON DROGON
That was it, thanks!

When I only changed the values of x and y I assumed the average would change when I told it to print, but it did not, I had to renter the average equation again (with no changes) for it to actually spit out the proper response (3.5) after I changed x and y to 3.0 and 4.0 respectively.

Is there a reason for this conceptually in python and is it different for other coding languages?


Also storing values as integers. Will all Python programs do this? The guide I'm following didn't mention that it would round decimals, so I'm still having trouble grasping why it wants to do this.

maniacripper fucked around with this message at 18:30 on Apr 7, 2013

peepsalot
Apr 24, 2007

        PEEP THIS...
           BITCH!

maniacripper posted:

That was it, thanks!

When I only changed the values of x and y I assumed the average would change when I told it to print, but it did not, I had to renter the average equation again (with no changes) for it to actually spit out the proper response (3.5) after I changed x and y to 3.0 and 4.0 respectively.

Is there a reason for this conceptually in python and is it different for other coding languages?


Also storing values as integers. Will all Python programs do this? The guide I'm following didn't mention that it would round decimals, so I'm still having trouble grasping why it wants to do this.
If you use Python 3 it will behave as you had expected.

Or put "from __future__ import division" at the top of your script. To get that functionality without python 3.

http://www.python.org/dev/peps/pep-0238/

Many other languages, such as C for example, would do the same (return an integer when dividing two integers). It's just keeping the typing consistent. Although it is maybe more apparent in C since the typing is explicit. In Python 3 the "/" operator is changed to return a float even when the two inputs are integers, and "//" is used to return an int.

In Python you have to include the decimal point to force python to store the value as float in its internal representation. Try "type(3)" vs "type(3.0)" in your python console to see.

peepsalot fucked around with this message at 18:47 on Apr 7, 2013

Janitor Prime
Jan 22, 2004

PC LOAD LETTER

What da fuck does that mean

Fun Shoe
I'm trying to put myself in your shoes and maybe the behavior you're relating to is Excel. In Excel a formula can use other cells to obtain its values and can be stored in it's own cell. When the values in the other cells change it automatically refreshes. Excel was created because most programming languages don't behave like that and require quite a bit of practice before anyone can start doing something similar to what Excel provides out of the box.

What you're doing right now is called imperative programming inside of a Read–eval–print loop (REPL) where you define statements that change the program state. Because the interpreter executes those statements sequentially even though you modified a variable, it can't assume that you meant to repeat every line that was previously executed, YOU have to do that manually.

If you wanted to program in that style you can, using a text editor and doing the typical edit-compile-run loop that is prevalent in other languages like C or Java which don't have a nice REPL to play around with in real time.

Dominoes
Sep 20, 2007

maniacripper posted:

Should I bother with ver2 at all? Should I just start learning ver3?

I'm learning Python, and chose Python 3 because it's the newer, refined version. You're more likely to run into compatibility problems with external modules in Python 3, since most of the information and development resources available are for Python 2. Philosophically, I suggest using Python 3 - you'd be helping push the community to upgrade.

Dominoes fucked around with this message at 19:48 on Apr 7, 2013

ShoulderDaemon
Oct 9, 2003
support goon fund
Taco Defender

maniacripper posted:

When I only changed the values of x and y I assumed the average would change when I told it to print, but it did not, I had to renter the average equation again (with no changes) for it to actually spit out the proper response (3.5) after I changed x and y to 3.0 and 4.0 respectively.

Is there a reason for this conceptually in python and is it different for other coding languages?

Python is an "imperative" programming language, as are the vast majority of programming languages. This means that it reads as a sequence of commands which the computer will perform in order. You should read "x = 3" as "the variable x now takes the value 3" and emphatically not as "x is defined as 3". So, the code
code:
x = 3
y = 4
average = (x + y) / 2
x = 10
is read as something like "First, x takes the value 3 and y takes the value 4. Now compute (x + y) / 2, and store that value in average. Lastly, x takes the value 10." We haven't told the computer to update the value stored in average, so that remains as it was.

There are non-imperative programming languages that behave like "real" mathematics. They are most often referred to as "declarative" or "assignment-free" programming languages, depending on context, as they do not typically allow you to change the definition of a variable. Haskell is probably the most visible example of such a language nowadays.

As someone who has taught beginning programmers, personally I think that the whole imperative-assignment thing is one of the worst things to teach and that people with no prior experience tend to have much more success with declarative languages than imperative languages, but unfortunately easily 99% of programming in the real world is done in imperative languages so it's just something you'll need to get used to.

maniacripper posted:

Also storing values as integers. Will all Python programs do this? The guide I'm following didn't mention that it would round decimals, so I'm still having trouble grasping why it wants to do this.

This is an ugly artifact that results from a poor language design and the nature of numeric representations in computers.

See, integer math is a hell of a lot faster than decimal math in computers. This has always been true, and historically the difference was even greater than it is today. In addition, computers model decimal math using something called floating point numbers, which are very far from perfect; for example, on my machine the expression "0.1 + 0.2" gives the result "0.30000000000000004" because the floating point representation introduces error. So there's a strong historical precedent and pratical requirement in programming languages to make it easy to do strictly integer math, which is both fast and accurate.

And that's fine, but Python as a programming language wants to make its programs look simple, so it doesn't use any kind of explicit marker to indicate which expressions are supposed to be integer-only and which ones are supposed to use floating point. Instead, it uses a heuristic rule: if all the numbers in an expression are integers, then use integer math. Otherwise, use floating point math.

So "(3 + 4) / 2" results in "3". But "(3.0 + 4) / 2" results in "3.5". Same with "(3 + 4.0) / 2" and "(3 + 4) / 2.0". You can even write something like "(3 + 4 + 0.0) / 2". As long as at least one of the numbers is written as a decimal, Python will use floating point math. But if you write them all as integers, then Python will use integer math, which truncates the decimal part.

But note that if you write "(3 + 4) / 2 + 0.0" then you will get "3.0" as the result. Why? Because Python sees that the "(3 + 4) / 2" part of the expression is all integers, so it does integer math and truncates the decimal, and only then does it convert to floating point and add the 0.0, giving you a weird result.

Newer versions of Python fix this problem by always using floating point math when dividing normally, and making you have to explicitly ask for integer division with a separate operator.

evensevenone
May 12, 2001
Glass is a solid.

maniacripper posted:

That was it, thanks!

When I only changed the values of x and y I assumed the average would change when I told it to print, but it did not, I had to renter the average equation again (with no changes) for it to actually spit out the proper response (3.5) after I changed x and y to 3.0 and 4.0 respectively.

Is there a reason for this conceptually in python and is it different for other coding languages?


Also storing values as integers. Will all Python programs do this? The guide I'm following didn't mention that it would round decimals, so I'm still having trouble grasping why it wants to do this.


For your first question, that's generally something you will find to be true for most languages. What you have to understand is that each line is executed in sequence, and it won't go back and change the results of earlier lines because something happened later.

It's also important to understand what the equals sign means, it doesn't really mean "equals". What it means is "take the right side, evaluate it, use the results of that to create a new object, and name that new object whatever is on the left side"

So when you are doing "a = 3" what is really going on is it's saying "ok 3 looks like the value of an integer, I'm gonna make an integer object, and name it 'a' "

When you do "b = a + 1" it says "ok, I don't know how to add 1 to an object, so I'm gonna get the value of the object, which is 3, and add 1 to that. That's 4, which is an integer, so I'll make a new integer object and name that 'b' ".

But then if you go and change "a", it's not going to change "b", and if you ask it for "b", it's still going to be "4". That's because it only does things once, it's not like Excel where it constantly re-updates all of the formulas.

For the second question, in python 2 dividing one int by another produces another int. It's just a rule. It makes sense for some things, less sense for other.

peepsalot
Apr 24, 2007

        PEEP THIS...
           BITCH!

ShoulderDaemon posted:

You can even write something like "(3 + 4 + 0.0) / 2". As long as at least one of the numbers is written as a decimal, Python will use floating point math. But if you write them all as integers, then Python will use integer math, which truncates the decimal part.
This is not the most accurate way of saying it. If I do "(3 + 4) / 2 + 0.0", this satisfies "at least one of the numbers is written as decimal", but would not use floating point for the entire calculation. Each arithmetic operator individually, executed in order of operator precedence, will return either int (if all inputs to the operation are int), or float if otherwise.

maniacripper
May 3, 2009
STANNIS BURNS SHIREEN
HIZDAR IS THE HARPY
JON GETS STABBED TO DEATH
DANY FLIES OFF ON DROGON
So much good information, I'm gonna wrap my head around this stuff when I get home from work, how exciting!

evensevenone
May 12, 2001
Glass is a solid.
Yeah, if you can get a good grasp on how assignments work and objects are created / referred to, it will really help as you learn the rest of the language, especially once you start writing your own functions and classes.

A couple of functions to help you understand this are type() and id(). type(x) will tell you what type an object is, and id(x) will tell you pythons internal id for that object.
Try things like:
Python code:
x = 5
y = x
print id(x) , id(y)
print x ,  y
x = 6
print id(x) , id(y)
print x ,  y

Dren
Jan 5, 2001

Pillbug
Since no one mentioned it, 2.7.4 is out. http://www.python.org/download/releases/2.7.4/

Winkle-Daddy
Mar 10, 2007
Hey all,

I was playing with the idea of writing a python server to be used for controlling software via a client GUI or web browser. This led me to play around with BaseHTTPRequestHandler, in that I subclass it to make a server and do my various handling in there. But, I've run into a bit of a hic-up and was hoping someone could point me in the direction to fix it the "right" way.

Here's the main fs_server class:

code:
class fs_server():
	def __init__(self):
		self.IS_RUNNING = "yes"
	def startServer(self):
		try:
			if(self.IS_RUNNING == "yes"):
				self.server = HTTPServer(('',8968),myServer)
				print("Server started...")
				self.server.serve_forever()
			else:
				self.server.socket.close()
		except KeyboardInterrupt:
			print '^C received, shutting down server!'
			self.server.socket.close()
	def shutdown(self):
		# Disable traceback, if we send
		# a kill to the server, it does not 
		# look good to the user!
		sys.tracebacklimit = 0
		self.server.socket.close()
Based on my comments, I think you can see what I'm doing here... I instance the server simply by doing something like "fsServer = fs_server()" then "fsServer.startServer()".

Now, if a specific request is sent to the "myServer" class, I want to kill the server. I've been trying to do it with fsServer.shutdown(), but when I send the "kill" request, this is what happens:

code:
localhost - - [08/Apr/2013 09:02:54] "GET /ABXz8fr210IFSczfW2ffazf HTTP/1.1" 200 -
KILL SESSION KEY SENT!
Traceback (most recent call last):
  File "main.py", line 70, in <module>
    theServer.startServer()
  File "main.py", line 55, in startServer
    self.server.serve_forever()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 225, in serve_forever
    r, w, e = select.select([self], [], [], poll_interval)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 444, in fileno
    return self.socket.fileno()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 170, in _dummy
    raise error(EBADF, 'Bad file descriptor')
socket.error: [Errno 9] Bad file descriptor
I'm sure there's some other method I should be calling instead, but I'm kinda new to making server stuff so some help would be appreciated!

My hack right now is to just do sys.tracebacklimit = 0 right before I call that to hide the error...but, that's not a good way of doing it. I want to do better.

EDIT: Might as well ask the next question, too! I'm building a direct peer to peer file sharing app that AES encrypts the files to be transferred. Part of using this server like this is to send a request for a specific file, then the server will send back a unique session ID, a new port and will move the file requested to a "transmit" folder where it will be AES encrypted and streamed as a base_64 encoded string from the server to the client. Since all peers will be running the server code, it would be pretty useless if for every transfer you had to go set up port forwarding.

I've heard there are some python upnp libraries for requesting a port forward to/from a specific address. Though, I've not seen any that seem easy enough to do for all operating systems and all upnp devices. Is there any python standard for doing this? If someone has worked with upnp before, what was your experience and could you share some example code?

Winkle-Daddy fucked around with this message at 17:33 on Apr 8, 2013

Dren
Jan 5, 2001

Pillbug
Have you considered a python webserver like bottle or flask?

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
Don't use BaseHTTPServer. It's bad. Twisted would be my recommendation.

Dren posted:

Have you considered a python webserver like bottle or flask?

These aren't web servers. They're web frameworks.

Winkle-Daddy
Mar 10, 2007

Suspicious Dish posted:

Don't use BaseHTTPServer. It's bad. Twisted would be my recommendation.

Given the very simple scope my application has, what is the benefit of Twisted? I think I've seen some Twisted + upnp firewall punching examples, so I'm very interested in moving to it. I'm just looking for a bit more information on why it would better suit my needs.

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
So, first, BaseHTTPServer is just broken, and you're simply looking at some of its brokenness. Twisted is, as far as I know, the only non-broken HTTP server. It's weird in places (see earlier in the thread, though I've since reversed my decision on having trailing slashes be accepted by default — but it's still too difficult to turn that on by default. It also doesn't handle rendering resources from Deferreds, which sucks), but it sounds like what you're looking for.

Winkle-Daddy
Mar 10, 2007

Suspicious Dish posted:

So, first, BaseHTTPServer is just broken, and you're simply looking at some of its brokenness. Twisted is, as far as I know, the only non-broken HTTP server. It's weird in places (see earlier in the thread, though I've since reversed my decision on having trailing slashes be accepted by default — but it's still too difficult to turn that on by default. It also doesn't handle rendering resources from Deferreds, which sucks), but it sounds like what you're looking for.

That sucks, I really liked how it took a few dozen lines of code to set up an arbitrary local web server. I appreciate your input and will take another, closer, look at twisted.

accipter
Sep 12, 2003

Suspicious Dish posted:

So, first, BaseHTTPServer is just broken, and you're simply looking at some of its brokenness.

I seem to recall that Python 3 fixed these issues. Is this correct?

Suspicious Dish
Sep 24, 2011

2020 is the year of linux on the desktop, bro
Fun Shoe
I wouldn't know. I don't use Python 3, and I don't use BaseHTTPServer.

Winkle-Daddy
Mar 10, 2007
I'm using Python 2.7 due to some dependencies which have yet to be ported to 3. So even if it is fixed (which is good to hear!) it doesn't help me for this project. Has anyone used twisted + http://coherence.beebits.net/browser/trunk/Puncher/puncher/base.py for upnp punching? I want to ensure that if I build with the various python packaging software I'm not going to horribly break anything.

My Rhythmic Crotch
Jan 13, 2011

Tornado has a built in webserver. It's not very fast, but it works.

Tornado also supports websockets, and I was developing a webchat server using Tornado and Flask mooshed together, just because it was kinda fun. Ultimately pointless though.

OnceIWasAnOstrich
Jul 22, 2006

I've had very good luck with ZeroMQ for this kind of stuff via pyzmq when I wanted to separate out communication from the actual webserver.

ElGroucho
Nov 1, 2005

We already - What about sticking our middle fingers up... That was insane
Fun Shoe
Can someone please tell why I am such a disaster at unzipping files in python?

code:
with zipfile.ZipFile('Software_Tokens.zip', 'r') as z:
    z.extractall()
works perfectly, but

code:
z = zipfile.Zipfile('Software_Tokens.zip', 'r')
gives me a 'module' object has no attribute 'Zipfile' error.

Using python 2.7.

Hot Dog Day #42
Jun 17, 2001

Egbert B. Gebstadter
You didn't capitalize the f in the second example.

ElGroucho
Nov 1, 2005

We already - What about sticking our middle fingers up... That was insane
Fun Shoe
Ugh, are you serious... Thank you.

accipter
Sep 12, 2003

ElGroucho posted:

Ugh, are you serious... Thank you.

Don't worry, we have all made mistakes like that.

Experto Crede
Aug 19, 2008

Keep on Truckin'
Is it possible to use pipes when using subprocess.call? So say I wanted to run a command and then run grep on its output and take the results of grep and make use of that in python, is that doable within a single subprocess.call instance? As it stands, I, for example want to run "lspci | grep 00:1a.0", but it seems subprocess.call is taking the pipes as part of the argument of lspci.

Anything I can do about this?

Scaevolus
Apr 16, 2007

Experto Crede posted:

Is it possible to use pipes when using subprocess.call? So say I wanted to run a command and then run grep on its output and take the results of grep and make use of that in python, is that doable within a single subprocess.call instance? As it stands, I, for example want to run "lspci | grep 00:1a.0", but it seems subprocess.call is taking the pipes as part of the argument of lspci.

Anything I can do about this?

Do the grep in python -- lspci doesn't produce that much output.

code:
import subprocess
lspci_lines = subprocess.check_output(['lspci']).splitlines()
lspci_lines = [line for line in lspci_lines if '00:1a.0' in line]
If you really want to do pipes:

code:
import subprocess
lspci_proc = subprocess.Popen(['lspci'], stdout=PIPE)
grep_proc = subprocess.Popen(['grep', '00:1a.0'],stdin=lspci_proc.stdout, stdout=PIPE)
out, err = grep_proc.communicate()

chemosh6969
Jul 3, 2004

code:
cat /dev/null > /etc/professionalism

I am in fact a massive asswagon.
Do not let me touch computer.
I need to use the cx_oracle module on windows 7 but I get an error that comes up when python isn't run as an admin. I can choose to have it run as an admin but then I have to enter my user/pass every time I want to run python, which got old really fast. Apparently is something new that came up with 7 and I haven't had much luck finding anyone else really running into the problem.

I can also remote into another server, that's some other version of windows, where I don't have the issue but that sucks as an option too.

Is there some equivalent module where I won't run into this issue or am I stuck with crappy options?

Dren
Jan 5, 2001

Pillbug

Experto Crede posted:

Is it possible to use pipes when using subprocess.call? So say I wanted to run a command and then run grep on its output and take the results of grep and make use of that in python, is that doable within a single subprocess.call instance? As it stands, I, for example want to run "lspci | grep 00:1a.0", but it seems subprocess.call is taking the pipes as part of the argument of lspci.

Anything I can do about this?

code:
subprocess.check_output('lspci | grep 00:1a.0', shell=True)

Adbot
ADBOT LOVES YOU

BeefofAges
Jun 5, 2004

Cry 'Havoc!', and let slip the cows of war.

If you're going to use subprocess with shell=True, make sure to read up on all the potential security problems that go along with it.

  • Locked thread