Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›3 »

vikingstrike: Sep 23, 2007; whats happening, captain

Just set up an alias so python points where you want it to.

# ¿ Sep 9, 2011 18:05

Adbot: ADBOT LOVES YOU

# ¿ Apr 27, 2024 13:02

vikingstrike: Sep 23, 2007; whats happening, captain

Why don't you just do the matrix algebra instead of using a class? Seems overkill. Make your x vectors a matrix and then multiply.

# ¿ Sep 18, 2011 02:35

vikingstrike: Sep 23, 2007; whats happening, captain

I know what the code is trying to do from school, but the only thing I can imagine he is doing is not creating his dependent/independent variable matrices correctly for input. I noticed in earlier code he posted that he might not have been using the X matrix correctly.

Aside from that, I think he should check the answers in a stat program that has OLS built in to make sure he isn't making errors in his calculations when checking.

Btw, those variable names do make more sense if you've done the stats/math in school.

# ¿ Sep 22, 2011 15:02

vikingstrike: Sep 23, 2007; whats happening, captain

I'm not defending the choice of variable names, but I'm just saying that if you understand the context of the programming it's not too bad. Representations like 'e' for residuals is how the math is written in textbooks/teachers everywhere.

Also, what foobat says holds a lot of truth. When you get people that are learning to code on the fly and then they start teaching other people it doesn't lead to happy places. Just lots of confusion.

vikingstrike fucked around with this message at 17:08 on Sep 22, 2011

# ¿ Sep 22, 2011 17:05

vikingstrike: Sep 23, 2007; whats happening, captain

We should all be happy that it was nobs and not just n.

# ¿ Sep 23, 2011 15:12

vikingstrike: Sep 23, 2007; whats happening, captain

So, I'm working on a project that requires me to fill out a Javascript form over and over and over again (basically, I enter a patent number and it gives me results). I would love to be able to automate this, so I can regex the results and store what I need in an output file. I checked out the mechanize module, but it has no Javascript support. Do any of you have experience doing things like this? Any modules that I should check out? I can post the source of the website, if needed.

# ¿ Nov 22, 2011 19:40

vikingstrike: Sep 23, 2007; whats happening, captain

Kim Jong III posted:

The source of the website might be helpful.

I'm guessing the Javascript does an AJAX POST somewhere and puts the data back into the page.

You might be able to POST directly to the AJAX endpoint using urllib2 or restkit or the like.

You've then got some options on the output... basic string parsing, regex, or use BeautifulSoup if it's nasty HTML and you need good screen scraping.

As I mentioned before the last time I posted about batch-scraping webpages: don't be an rear end in a top hat to the site owner. Throttle your bot. I always try to be extra nice and give at least 10-15 seconds between requests.

Yeah, I don't want to be an rear end in a top hat at all. Actually most of the time I scrape websites, I use 15-20 second buffers.

Here's a paste bin of the source. Sorry for the large spaces, it's how I copied it off the page. I need to be able to include text in the search box, check a box and then change the search date range.

http://pastebin.com/T4c8Ch5P

# ¿ Nov 22, 2011 21:04

vikingstrike: Sep 23, 2007; whats happening, captain

Kim Jong III posted:

Oof, that's a ton of code.

I see LexisNexis so I'm assuming everything is paywalled, or I'd say link us the webpage itself & let's try it out. I don't see where onSearch() is defined, which looks to be the function that kicks off the search, so that's probably buried in one of the many referenced JS files.

My suggestion here - rather than trying to pore over hundreds of lines of nasty HTML, try to capture the submit at the source. For this, Firebug is your friend. Install it, go to your page, enable the Net panel, and submit the form. I'm betting you'll see a POST in the XHR section. That POST will probably have a ton of data, but will have your searched text, checked box, and date range.

I'd start there. See if you can deconstruct the POST, and reconstruct it without using the webpage. curl is really good at this -- use -X POST to send a POST (if that's the right verb, of course!), -d for data, and if the site authenticates with cookies, -b is your friend.

Thanks for the information. This is exactly the type of feedback I was looking for! Just hope I can get it working :smith:

# ¿ Nov 23, 2011 14:53

vikingstrike: Sep 23, 2007; whats happening, captain

Victor Vermis posted:

Nope! Googled that, and slipped an empty file titled __init__.py into the folder my scripts are in but still no luck.

Looking through google matches explaining __init___ , they all gloss over the process of saving a script.
"You can make a script! Type your code, save it as .py, and then use the import command.."
There's no explanation of where and why you would stash these scripts or what else needs to be done to ensure Step 1 (create) works with Step 2 (import).

As an experiment, tried importing a script that was part of the install, and also located in the Scripts folder.
code:
from byext import *
And received the same "Error no module named byext".

You may have touched on this and I missed it but is byext.py in the same folder as the script you are running? Obviously there are ways around this but for a beginner it would probably be the easiest solution.

# ¿ Nov 24, 2011 01:31

vikingstrike: Sep 23, 2007; whats happening, captain

Victor Vermis posted:

Yes, byext.py and whatever script I'm trying to run, say atest.py, are both located in C:\Python25\Tools\Scripts

Am I mistaken thinking that entering the following code into the shell would then activate all of the variable assignments and whatnot within atest.py?
code:
from atest import *
^ this command being the specific problem at hand. I see it used as an example and am instructed to use it, but I have no idea how or why it works (or in this case doesn't work), unlike Function Definitions and Type and Assignments and Operands and everything else that has a dedicated section within the text.

When you are saying "entering the following code into the shell", do you mean that in your terminal you are typing "from byext import *" before running "python atest.py"? If so, you need to put the "from byext import *" at the top of the "atest.py" file. Then just open your terminal and type "python atest.py" to run it.

# ¿ Nov 24, 2011 02:33

vikingstrike: Sep 23, 2007; whats happening, captain

From test1.py

code:

poo = 'message'

def returnPoo():
	return "poo"

From test2.py

code:

from test1 import *

print returnPoo()

print poo

And when I run it:

code:

Randy:Desktop $ python test2.py
poo
message

I never use IDLE, so I have no idea if that's causing you trouble. But, from what I am understanding you're trying to do, the above example shows you what's going on. If you do just "import test1", then you would just change the calls to test1.returnPoo() and test1.poo.

# ¿ Nov 24, 2011 04:57

vikingstrike: Sep 23, 2007; whats happening, captain

I do have to say it was quite enjoyable to write a function called returnPoo(). Ha.

# ¿ Nov 24, 2011 14:41

vikingstrike: Sep 23, 2007; whats happening, captain

Kim Jong III posted:

POST STUFF....FIREBUG

Just wanted to follow up and say that your advice worked perfectly for me. Thanks again!

# ¿ Nov 27, 2011 21:15

vikingstrike: Sep 23, 2007; whats happening, captain

I think you just need to do:

print dave

and

print "Hello, %s" % dave

At least, if I'm understanding what you're trying to do.

# ¿ Dec 5, 2011 21:50

vikingstrike: Sep 23, 2007; whats happening, captain

I usually just use strip() along with readlines() to store it right from the beginning.

input_lines = [line.strip() for line in input.readlines()]

I have no idea why I do this. Would it be better to just leave it alone and do what he is?

# ¿ Feb 11, 2012 02:51

vikingstrike: Sep 23, 2007; whats happening, captain

Well I guess you learn things everyday. Thanks for the feedback.

# ¿ Feb 11, 2012 07:32

vikingstrike: Sep 23, 2007; whats happening, captain

I'm not at my main computer to post code, but try using the cookie manager and build this into the opener. I'm not sure what properties are on that password manager but you probably just need cookie support.

# ¿ Feb 16, 2012 06:50

vikingstrike: Sep 23, 2007; whats happening, captain

What's wrong with urllib2? Between it and urllib I've never really run into issues. Does requests have special functionality that the other two don't offer?

# ¿ Feb 16, 2012 07:19

vikingstrike: Sep 23, 2007; whats happening, captain

I'm not sure if this has been posted, but I just found this and it's pretty awesome:

http://fonnesbeck.github.com/ScipySuperpack/

quote:

This shell script will install recent 64-bit builds of Numpy (2.0) and Scipy (0.11), Matplotlib (1.2), iPython (0.13), Pandas (0.7), as well as PyMC (2.2 alpha) for OS X 10.7 (Lion) on Intel Macintosh. All builds are based on recent development code from each package, which means though some bugs may be fixed and features added, they also may be more unstable than the official releases[1]. Distributing them together should improve interoperability, since the supporting packages (Scipy, Matplotlib, PyMC) were all built against the accompanying build of Numpy. I encourage all users to run the appropriate unit tests on each package to ensure that any extant issues do not affect your work. Please report any errors that may be the result of build issues.

These packages were compiled on OS X 10.7.3 using Apple�s Python 2.7.1, gFortran 4.2.3 and LLVM-GCC 4.2.1 (build 5658). To avoid compatibility issues, the installer also optionally downloads and installs the gFortran compiler that is compatible with Xcode 4.2. Before you install the Superpack, ensure that Xcode 4.2 is installed on your system.

The packages are updated approximately every two weeks. Check the repository for the build date.

# ¿ Feb 27, 2012 23:42

vikingstrike: Sep 23, 2007; whats happening, captain

I've never programmed those problems before but if you know the math, writing functions should be pretty straight forward. I'd use SciPy/NumPy. I'd bet with a little searching you could find example code online that probably does what you need.

# ¿ Feb 28, 2012 13:08

vikingstrike: Sep 23, 2007; whats happening, captain

I feel dumb asking this, but can someone explain what all is going on here? (it's from the link above)

code:

f = lambda x: x and x * f(x - 1) or 1

I've never really used lambda functions before, which may be part of the problem.

# ¿ Mar 8, 2012 21:54

vikingstrike: Sep 23, 2007; whats happening, captain

Plorkyeran posted:

f = lambda x: ... is (in this case) equivalent to def f(x): return .... x and y returns y if x evaluates to true, and false otherwise (which is equivalent to boolean and if they're both bools, and more useful if they aren't). x or y returns x if it's true, and y otherwise. As such, the body of the lambda is equivalent to
code:
if x:
    return x * f(x - 1)
else:
    return 1

Thanks for this!

# ¿ Mar 9, 2012 00:23

vikingstrike: Sep 23, 2007; whats happening, captain

If it's an input file, I would personally shove all of that on a single line if it needs to be. I only worry about "pythonic" crap inside my .py files.

I'd probably do something like:

code:

import re

try:
    db_lines = [re.sub(';', '', line) for line in open('dbfile.whatevs', 'r')]
except IOError:
    print "poo poo is broken"

# ¿ Mar 26, 2012 21:34

vikingstrike: Sep 23, 2007; whats happening, captain

tef posted:

(fwiw most of the python developers I know on osx use a linux vm, or connect to a linux box, instead of the slow tortuous painful death that is installing packages on osx for python)

I've actually thought about doing this, too. Glad to know I'm not crazy. yum is so awesome and it pains me that OS X has no complement that works nearly as well.

# ¿ Mar 28, 2012 13:29

vikingstrike: Sep 23, 2007; whats happening, captain

In the software os x thread there are people that mention brew fucks with the permissions on /usr/bin and other folders. Any truth to this?

Edit: oh poo poo with textmate 2's rmate script the VM route may be the perfect ticket.

vikingstrike fucked around with this message at 15:06 on Mar 28, 2012

# ¿ Mar 28, 2012 15:02

vikingstrike: Sep 23, 2007; whats happening, captain

Sailor_Spoon posted:

brew recommends that you do something awful like chmod 777 /usr/local/bin. You don't have to, though. They just seem to think that requiring users to use sudo to install system packages is somehow a bad thing.

Ahh, thanks for the clarification! Yeah, that advice is dumb as poo poo. Jesus, that's really bad.

Another reason I asked was that I installed it once, checked my folder perms, and wondered what the hell people were talking about.

# ¿ Mar 28, 2012 16:07

vikingstrike: Sep 23, 2007; whats happening, captain

JetsGuy posted:

So I got into this whole discussion today with someone about high-level lanuages vs low-level languages. The meat of the problem is the guy is coding some fairly involved numerical integrals that have to literally be run thousands of times (lots of data!). The problem is that he originally coded it in Mathematica, but is now moving it cpp because it runs entirely too slow on Mathematica.

I understand one of the biggest "pluses" (nyuk nyuk) regarding cpp is its speed, but I'm not entirely convinced you couldn't code something like that in python. IF coded well, I bet you could do it without much difference in speed. I guess the counter arguement would be that in cpp you'd not have to be as creative with coding as you do with cpp to get it to run fast.

Then again, the biggest mistake I think people make is they try to optimize too early. Design and write your code, get it to work right. THEN try to slim down time if you don't like how fast it works. Then again, I'm thinking from a scientific programming angle, so maybe this is a laughable perspective for a py developer.

In general, which py libraries do you use most often? I'm assuming that you're using nump/scipy, at least. I do Matlab work with people at school, but at home, I've been doing personal work in python and am wondering if there is a nice package I'm missing. I pretty much just use the ScipySuperpack.

# ¿ Mar 28, 2012 16:56

vikingstrike: Sep 23, 2007; whats happening, captain

JetsGuy posted:

I keep telling myself that I have to learn cpp so I can get a REAL JOB (TM) outside of scientific research, but I just am exhausted when I go home...

I feel you on this. A lot of the people that I run into use cpp. "It's low level, so I can do this and that and it's faster!" Whereas, I generally care more about the readability of my code and being able to develop quickly.

I believe that you do more numerical work than me (although, I've used numpy, matplotlib, and scipy from your list, no need for the astrophysics stuff, i really enjoy pandas too). A lot of the data work I do involves strings and manipulating, searching, and parsing, which I am always looking for techniques to improve on. Like, I'll have a file with 2,000,000+ rows that I need to basically grep quickly, store results, etc.

# ¿ Mar 28, 2012 17:24

vikingstrike: Sep 23, 2007; whats happening, captain

Auditore posted:

I'm sorry, I don't understand. The quotation marks don't change anything.

Yes, they do. Use the code from the post above and run type() on the answers to see it.

# ¿ Mar 29, 2012 12:19

vikingstrike: Sep 23, 2007; whats happening, captain

JetsGuy posted:

I had to reinstall python completely today. gently caress macports forever. That poo poo is an insidious disease that really fucks your system.

-m32? gently caress YOU, I KNOW BETTER

NO REFUSE TO PAY ATTENTION TO ANY OTHER BUILDS! EVERYTHING MUST BE IN THE OPT FOLDER OR IT DOESN'T EXIST!

Seriously, I had a compiler issue with perl because for whatever reason, heasoft was using the macports perl (which is poo poo, perl is poo poo).

So I had enough of macports being a bitch, and uninstalled the whole thing. This meant having to install python and ipython (etc etc) again, but it was remarkably pleasant how easy it was this time around.

I'm not looking forward to doing it when I get home on the Lion machine.

Not sure what all you use module wise, but look at SciPySuperpack. It's a bash script that will build numpy, scipy, matplotlib, statsmodels, pymc, pandas, and ipython against the Apple default Python and all you'll need additionally is Xcode.

Otherwise I use the python.org site to install 2.7 or use homebrew instead of macports to install it.

# ¿ Apr 6, 2012 22:44

vikingstrike: Sep 23, 2007; whats happening, captain

Might be a stupid follow-up but are you looking for plotting libraries or data manipulation type libraries? Or neither and I will leave this to an expert.

# ¿ Apr 17, 2012 20:27

vikingstrike: Sep 23, 2007; whats happening, captain

I really enjoy iPython and at times find myself using it over the bash shell to get work done. (I can think in Python easier than bash on the fly.)

I would recommend checking out the docs because there are some pretty cool features that you can use within it.

# ¿ Apr 17, 2012 21:50

vikingstrike: Sep 23, 2007; whats happening, captain

Suspicious Dish posted:

I do not like IPython. It made me run in circles for a while trying to figure out why my code was broken, until I realized that it was IPython that broke my code, and my code was fine.

I use bpython when I need a somewhat fancy interpreter. It doesn't work on Python 3, or Windows, though.

Out of curiosity now, what was iPython doing? I usually just use it for one off things, but I'll move to something else if it's broken.

# ¿ Apr 17, 2012 22:03

vikingstrike: Sep 23, 2007; whats happening, captain

Looking to use some basic GUI tools in a program I'm writing. Never done any GUI programming and don't really have the energy to learn something very difficult. I've found easygui online and it looks to be pretty straight forward. Any opinions on easygui, or are there any better alternatives?

I guess that I could list some of the things I'm looking to do:
- Allow the user to use the normal OS X/Windows directory browser to choose a folder
- Allow the user to select certain items out a displayed list
- Capture basic strings from the user, like "name", "author", etc.

Well, it looks like EasyGUI doesn't allow basic poo poo like "I want the window size to be X", so yeah, I don't think this will work. Tkinter is in the standard library, which is nice. Is this the easiest way to go? Cross-platform compatibility and easy-of-use are important to me, but this may be a trade off type situation.

vikingstrike fucked around with this message at 19:08 on Apr 18, 2012

# ¿ Apr 18, 2012 18:55

vikingstrike: Sep 23, 2007; whats happening, captain

deimos posted:

Kinda late for this, but if you don't care about adding a dependency I love the speed of lxml, and then there's this:
http://lxml.de/FAQ.html#how-can-i-map-an-xml-tree-into-a-dict-of-dicts

According to: http://priestc.github.com/django-easydump/ (which I got from your link) you have to: pip install django-easydump

As a side question, anyone know a good review on math subjects like linear algebra and diff eq?

I was looking at that markov chains pdf posted earlier and realized how much I had forgotten.

Khan Academy has some pretty good, short math/stats tutorial videos.
http://www.khanacademy.org/#linear-algebra
http://www.khanacademy.org/#differential-equations
http://www.khanacademy.org/#probability

# ¿ Apr 20, 2012 17:37

vikingstrike: Sep 23, 2007; whats happening, captain

Reading this blog was really helpful in getting accustomed to the standard library:
http://www.doughellmann.com/PyMOTW/

I still use it from time to time for quick reference.

# ¿ May 8, 2012 17:05

vikingstrike: Sep 23, 2007; whats happening, captain

For grabbing the current working directory, you can use os.getcwd().

# ¿ May 22, 2012 17:05

vikingstrike: Sep 23, 2007; whats happening, captain

JetsGuy posted:

Yeah, I had a feeling there was an easier way to grab that, thanks. I still have to run shell scripts though :/.

Yeah, I don't know of a better way to do that. I use subprocess.check_output(), too.

# ¿ May 22, 2012 17:30

vikingstrike: Sep 23, 2007; whats happening, captain

You could always use Python to create the bash scripts, and then run them from another bash script.

# ¿ May 22, 2012 18:11

Adbot: ADBOT LOVES YOU

# ¿ Apr 27, 2024 13:02

vikingstrike: Sep 23, 2007; whats happening, captain

^^Kind of beaten. His way works too.

I think it would be a little easier to do something like:

Python code:

>>> students = {}
>>> students["Alex"] = {'test1':85}
>>> students["Anne"] = {'test1':90}
>>> students
{'Anne': {'test1': 90}, 'Alex': {'test1': 85}}
>>> students["Alex"]["test1"]
85
>>> students["Anne"]["test1"]
90

# ¿ May 25, 2012 20:12

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›3 »