|
Whichever method you are using to download the csv, save it into a string and then use cStringIO to turn that string into a file handle which you feed to csv.reader().
|
# ? Apr 6, 2013 02:07 |
|
|
# ? May 8, 2024 18:38 |
|
Dominoes posted:Another CSV / memory vs drive write question. If I want to download a CSV from a url, is there a way I can put the file information directly into a variable in memory? I've tried many combinations using list(), read(), csv.reader() etc, but can't find a working solution that skips writing the csv to disk, and reading it. It doesnt work because reader expects file handle not string. Maybe you can use StringIO to makes this easy like this: Python code:
|
# ? Apr 6, 2013 02:09 |
|
Dominoes posted:For example, this seems like it should do what I'm asking, but doesn't. Did you try: Python code:
|
# ? Apr 6, 2013 02:13 |
|
I'm trying to use subprocess.call to run a wget command which uses a variable as part of it. The code is: Python code:
Either way, any advice?
|
# ? Apr 6, 2013 18:18 |
|
Python code:
Python code:
NOTinuyasha fucked around with this message at 18:40 on Apr 6, 2013 |
# ? Apr 6, 2013 18:31 |
|
NOTinuyasha posted:Is the straight answer. But really... And if you're only able to use built in libraries: Python code:
|
# ? Apr 6, 2013 18:35 |
|
BigRedDot posted:Hi guys, I just wanted to throw out that the videos of tutorials and talks from our latest PyData conference are up on vimeo now: Did you do the luigi talk? How does hadoop attach a module thing work? Does hadoop have a provision for it or are you sending bytecode somehow? (I am a hadoop noob).
|
# ? Apr 6, 2013 23:20 |
|
Dren posted:Did you do the luigi talk? How does hadoop attach a module thing work? Does hadoop have a provision for it or are you sending bytecode somehow? (I am a hadoop noob). No, sorry I shouldn't have been so coy. I gave the Numpy tutorial.
|
# ? Apr 7, 2013 05:24 |
|
So I'm starting to learn Python (2.7.3) and I have absolutely zero programming/scripting experience. Should I bother with ver2 at all? Should I just start learning ver3? Either way I'm doing some exercises and I can't get it to math properly. It's a simple math operation where I want to print the average of 2 values. The program looks like this: x = 3 y = 4 average = (x + y) / 2 print(average) 3 <----- this poo poo right here is wrong wrong wrong! What am i doing wrong?
|
# ? Apr 7, 2013 18:20 |
|
It's guessing that you want to store 3 and 4 as ints, and rounding down the result so it's also an int. Try 3.0 and 4.0.
|
# ? Apr 7, 2013 18:23 |
|
That was it, thanks! When I only changed the values of x and y I assumed the average would change when I told it to print, but it did not, I had to renter the average equation again (with no changes) for it to actually spit out the proper response (3.5) after I changed x and y to 3.0 and 4.0 respectively. Is there a reason for this conceptually in python and is it different for other coding languages? Also storing values as integers. Will all Python programs do this? The guide I'm following didn't mention that it would round decimals, so I'm still having trouble grasping why it wants to do this. maniacripper fucked around with this message at 18:30 on Apr 7, 2013 |
# ? Apr 7, 2013 18:26 |
|
maniacripper posted:That was it, thanks! Or put "from __future__ import division" at the top of your script. To get that functionality without python 3. http://www.python.org/dev/peps/pep-0238/ Many other languages, such as C for example, would do the same (return an integer when dividing two integers). It's just keeping the typing consistent. Although it is maybe more apparent in C since the typing is explicit. In Python 3 the "/" operator is changed to return a float even when the two inputs are integers, and "//" is used to return an int. In Python you have to include the decimal point to force python to store the value as float in its internal representation. Try "type(3)" vs "type(3.0)" in your python console to see. peepsalot fucked around with this message at 18:47 on Apr 7, 2013 |
# ? Apr 7, 2013 18:37 |
|
I'm trying to put myself in your shoes and maybe the behavior you're relating to is Excel. In Excel a formula can use other cells to obtain its values and can be stored in it's own cell. When the values in the other cells change it automatically refreshes. Excel was created because most programming languages don't behave like that and require quite a bit of practice before anyone can start doing something similar to what Excel provides out of the box. What you're doing right now is called imperative programming inside of a Read–eval–print loop (REPL) where you define statements that change the program state. Because the interpreter executes those statements sequentially even though you modified a variable, it can't assume that you meant to repeat every line that was previously executed, YOU have to do that manually. If you wanted to program in that style you can, using a text editor and doing the typical edit-compile-run loop that is prevalent in other languages like C or Java which don't have a nice REPL to play around with in real time.
|
# ? Apr 7, 2013 18:52 |
|
maniacripper posted:Should I bother with ver2 at all? Should I just start learning ver3? I'm learning Python, and chose Python 3 because it's the newer, refined version. You're more likely to run into compatibility problems with external modules in Python 3, since most of the information and development resources available are for Python 2. Philosophically, I suggest using Python 3 - you'd be helping push the community to upgrade. Dominoes fucked around with this message at 19:48 on Apr 7, 2013 |
# ? Apr 7, 2013 18:55 |
|
maniacripper posted:When I only changed the values of x and y I assumed the average would change when I told it to print, but it did not, I had to renter the average equation again (with no changes) for it to actually spit out the proper response (3.5) after I changed x and y to 3.0 and 4.0 respectively. Python is an "imperative" programming language, as are the vast majority of programming languages. This means that it reads as a sequence of commands which the computer will perform in order. You should read "x = 3" as "the variable x now takes the value 3" and emphatically not as "x is defined as 3". So, the code code:
There are non-imperative programming languages that behave like "real" mathematics. They are most often referred to as "declarative" or "assignment-free" programming languages, depending on context, as they do not typically allow you to change the definition of a variable. Haskell is probably the most visible example of such a language nowadays. As someone who has taught beginning programmers, personally I think that the whole imperative-assignment thing is one of the worst things to teach and that people with no prior experience tend to have much more success with declarative languages than imperative languages, but unfortunately easily 99% of programming in the real world is done in imperative languages so it's just something you'll need to get used to. maniacripper posted:Also storing values as integers. Will all Python programs do this? The guide I'm following didn't mention that it would round decimals, so I'm still having trouble grasping why it wants to do this. This is an ugly artifact that results from a poor language design and the nature of numeric representations in computers. See, integer math is a hell of a lot faster than decimal math in computers. This has always been true, and historically the difference was even greater than it is today. In addition, computers model decimal math using something called floating point numbers, which are very far from perfect; for example, on my machine the expression "0.1 + 0.2" gives the result "0.30000000000000004" because the floating point representation introduces error. So there's a strong historical precedent and pratical requirement in programming languages to make it easy to do strictly integer math, which is both fast and accurate. And that's fine, but Python as a programming language wants to make its programs look simple, so it doesn't use any kind of explicit marker to indicate which expressions are supposed to be integer-only and which ones are supposed to use floating point. Instead, it uses a heuristic rule: if all the numbers in an expression are integers, then use integer math. Otherwise, use floating point math. So "(3 + 4) / 2" results in "3". But "(3.0 + 4) / 2" results in "3.5". Same with "(3 + 4.0) / 2" and "(3 + 4) / 2.0". You can even write something like "(3 + 4 + 0.0) / 2". As long as at least one of the numbers is written as a decimal, Python will use floating point math. But if you write them all as integers, then Python will use integer math, which truncates the decimal part. But note that if you write "(3 + 4) / 2 + 0.0" then you will get "3.0" as the result. Why? Because Python sees that the "(3 + 4) / 2" part of the expression is all integers, so it does integer math and truncates the decimal, and only then does it convert to floating point and add the 0.0, giving you a weird result. Newer versions of Python fix this problem by always using floating point math when dividing normally, and making you have to explicitly ask for integer division with a separate operator.
|
# ? Apr 7, 2013 19:00 |
|
maniacripper posted:That was it, thanks! For your first question, that's generally something you will find to be true for most languages. What you have to understand is that each line is executed in sequence, and it won't go back and change the results of earlier lines because something happened later. It's also important to understand what the equals sign means, it doesn't really mean "equals". What it means is "take the right side, evaluate it, use the results of that to create a new object, and name that new object whatever is on the left side" So when you are doing "a = 3" what is really going on is it's saying "ok 3 looks like the value of an integer, I'm gonna make an integer object, and name it 'a' " When you do "b = a + 1" it says "ok, I don't know how to add 1 to an object, so I'm gonna get the value of the object, which is 3, and add 1 to that. That's 4, which is an integer, so I'll make a new integer object and name that 'b' ". But then if you go and change "a", it's not going to change "b", and if you ask it for "b", it's still going to be "4". That's because it only does things once, it's not like Excel where it constantly re-updates all of the formulas. For the second question, in python 2 dividing one int by another produces another int. It's just a rule. It makes sense for some things, less sense for other.
|
# ? Apr 7, 2013 19:07 |
|
ShoulderDaemon posted:You can even write something like "(3 + 4 + 0.0) / 2". As long as at least one of the numbers is written as a decimal, Python will use floating point math. But if you write them all as integers, then Python will use integer math, which truncates the decimal part.
|
# ? Apr 7, 2013 19:21 |
|
So much good information, I'm gonna wrap my head around this stuff when I get home from work, how exciting!
|
# ? Apr 7, 2013 19:31 |
|
Yeah, if you can get a good grasp on how assignments work and objects are created / referred to, it will really help as you learn the rest of the language, especially once you start writing your own functions and classes. A couple of functions to help you understand this are type() and id(). type(x) will tell you what type an object is, and id(x) will tell you pythons internal id for that object. Try things like: Python code:
|
# ? Apr 7, 2013 20:06 |
|
Since no one mentioned it, 2.7.4 is out. http://www.python.org/download/releases/2.7.4/
|
# ? Apr 8, 2013 13:53 |
|
Hey all, I was playing with the idea of writing a python server to be used for controlling software via a client GUI or web browser. This led me to play around with BaseHTTPRequestHandler, in that I subclass it to make a server and do my various handling in there. But, I've run into a bit of a hic-up and was hoping someone could point me in the direction to fix it the "right" way. Here's the main fs_server class: code:
Now, if a specific request is sent to the "myServer" class, I want to kill the server. I've been trying to do it with fsServer.shutdown(), but when I send the "kill" request, this is what happens: code:
My hack right now is to just do sys.tracebacklimit = 0 right before I call that to hide the error...but, that's not a good way of doing it. I want to do better. EDIT: Might as well ask the next question, too! I'm building a direct peer to peer file sharing app that AES encrypts the files to be transferred. Part of using this server like this is to send a request for a specific file, then the server will send back a unique session ID, a new port and will move the file requested to a "transmit" folder where it will be AES encrypted and streamed as a base_64 encoded string from the server to the client. Since all peers will be running the server code, it would be pretty useless if for every transfer you had to go set up port forwarding. I've heard there are some python upnp libraries for requesting a port forward to/from a specific address. Though, I've not seen any that seem easy enough to do for all operating systems and all upnp devices. Is there any python standard for doing this? If someone has worked with upnp before, what was your experience and could you share some example code? Winkle-Daddy fucked around with this message at 17:33 on Apr 8, 2013 |
# ? Apr 8, 2013 17:03 |
|
Have you considered a python webserver like bottle or flask?
|
# ? Apr 8, 2013 18:49 |
|
Don't use BaseHTTPServer. It's bad. Twisted would be my recommendation.Dren posted:Have you considered a python webserver like bottle or flask? These aren't web servers. They're web frameworks.
|
# ? Apr 8, 2013 18:51 |
|
Suspicious Dish posted:Don't use BaseHTTPServer. It's bad. Twisted would be my recommendation. Given the very simple scope my application has, what is the benefit of Twisted? I think I've seen some Twisted + upnp firewall punching examples, so I'm very interested in moving to it. I'm just looking for a bit more information on why it would better suit my needs.
|
# ? Apr 8, 2013 19:02 |
|
So, first, BaseHTTPServer is just broken, and you're simply looking at some of its brokenness. Twisted is, as far as I know, the only non-broken HTTP server. It's weird in places (see earlier in the thread, though I've since reversed my decision on having trailing slashes be accepted by default — but it's still too difficult to turn that on by default. It also doesn't handle rendering resources from Deferreds, which sucks), but it sounds like what you're looking for.
|
# ? Apr 8, 2013 19:15 |
|
Suspicious Dish posted:So, first, BaseHTTPServer is just broken, and you're simply looking at some of its brokenness. Twisted is, as far as I know, the only non-broken HTTP server. It's weird in places (see earlier in the thread, though I've since reversed my decision on having trailing slashes be accepted by default — but it's still too difficult to turn that on by default. It also doesn't handle rendering resources from Deferreds, which sucks), but it sounds like what you're looking for. That sucks, I really liked how it took a few dozen lines of code to set up an arbitrary local web server. I appreciate your input and will take another, closer, look at twisted.
|
# ? Apr 8, 2013 19:29 |
|
Suspicious Dish posted:So, first, BaseHTTPServer is just broken, and you're simply looking at some of its brokenness. I seem to recall that Python 3 fixed these issues. Is this correct?
|
# ? Apr 8, 2013 19:36 |
|
I wouldn't know. I don't use Python 3, and I don't use BaseHTTPServer.
|
# ? Apr 8, 2013 19:38 |
|
I'm using Python 2.7 due to some dependencies which have yet to be ported to 3. So even if it is fixed (which is good to hear!) it doesn't help me for this project. Has anyone used twisted + http://coherence.beebits.net/browser/trunk/Puncher/puncher/base.py for upnp punching? I want to ensure that if I build with the various python packaging software I'm not going to horribly break anything.
|
# ? Apr 8, 2013 20:08 |
|
Tornado has a built in webserver. It's not very fast, but it works. Tornado also supports websockets, and I was developing a webchat server using Tornado and Flask mooshed together, just because it was kinda fun. Ultimately pointless though.
|
# ? Apr 8, 2013 20:54 |
|
I've had very good luck with ZeroMQ for this kind of stuff via pyzmq when I wanted to separate out communication from the actual webserver.
|
# ? Apr 8, 2013 21:04 |
|
Can someone please tell why I am such a disaster at unzipping files in python?code:
code:
Using python 2.7.
|
# ? Apr 8, 2013 22:32 |
|
You didn't capitalize the f in the second example.
|
# ? Apr 8, 2013 22:34 |
|
Ugh, are you serious... Thank you.
|
# ? Apr 8, 2013 22:50 |
|
ElGroucho posted:Ugh, are you serious... Thank you. Don't worry, we have all made mistakes like that.
|
# ? Apr 8, 2013 22:57 |
|
Is it possible to use pipes when using subprocess.call? So say I wanted to run a command and then run grep on its output and take the results of grep and make use of that in python, is that doable within a single subprocess.call instance? As it stands, I, for example want to run "lspci | grep 00:1a.0", but it seems subprocess.call is taking the pipes as part of the argument of lspci. Anything I can do about this?
|
# ? Apr 9, 2013 17:44 |
|
Experto Crede posted:Is it possible to use pipes when using subprocess.call? So say I wanted to run a command and then run grep on its output and take the results of grep and make use of that in python, is that doable within a single subprocess.call instance? As it stands, I, for example want to run "lspci | grep 00:1a.0", but it seems subprocess.call is taking the pipes as part of the argument of lspci. Do the grep in python -- lspci doesn't produce that much output. code:
code:
|
# ? Apr 9, 2013 17:48 |
|
I need to use the cx_oracle module on windows 7 but I get an error that comes up when python isn't run as an admin. I can choose to have it run as an admin but then I have to enter my user/pass every time I want to run python, which got old really fast. Apparently is something new that came up with 7 and I haven't had much luck finding anyone else really running into the problem. I can also remote into another server, that's some other version of windows, where I don't have the issue but that sucks as an option too. Is there some equivalent module where I won't run into this issue or am I stuck with crappy options?
|
# ? Apr 9, 2013 18:04 |
|
Experto Crede posted:Is it possible to use pipes when using subprocess.call? So say I wanted to run a command and then run grep on its output and take the results of grep and make use of that in python, is that doable within a single subprocess.call instance? As it stands, I, for example want to run "lspci | grep 00:1a.0", but it seems subprocess.call is taking the pipes as part of the argument of lspci. code:
|
# ? Apr 9, 2013 18:09 |
|
|
# ? May 8, 2024 18:38 |
|
If you're going to use subprocess with shell=True, make sure to read up on all the potential security problems that go along with it.
|
# ? Apr 9, 2013 19:58 |