Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Eela6: May 25, 2007; Shredded Hen

[quote="huhu"]
Uneducated question - why does memory management matter? I understand why it would with something like an Arduino which has almost no memory but why would that matter on something like a computer with so much more memory? For context, I'm reading about arrays vs linked lists.
[\quote]

Speed, scaling, and professionalism.

Speed: Proper memory use is faster. If you manage your memory well, your CPU doesn't have to constantly load things in and out of the various caches. In many cases, proper memory management can lewd to 20-100x speedup of your code. If you're so wasteful of memory that you start having to constantly load things in and out of the drive, this is another couple orders of magnitude slower.

Scaling: Say you write a text-processing script of some kind that has to iterate over your material multiple times. So to make things easier, you turn it into a list, you load the whole thing into memory, then check the text. Most text files are under a megabyte, so who cares, right? This world fine for 99% of text corpuses. Then you get a 120gb text corpus - say, every single SA forum post. It's 50gb. You run your script, it locks your computer, you look silly.

Professionalism: You're a programmer. You should write good code, and take pride in it. You don't have to optimize every single thing, but if you always take that extra bit of time to make your code clean and efficient, it will make you a better coder. If you always do it right, it will become a matter of habit, and writing good code when it matters will become trivial. You want your default code to be good code. Memory management is only part of this, but people who are lazy and sloppy with memory are sloppy elsewhere. The buck stops with you.

Luckily, good memory use is easy in python. Use generator comprehensions rather than list comprehensions. On that note, don't return a list if you can yield the contents. Use the correct data structures. Write small functions, so that intermediate variables exit scope and are deleted by the GC. Read through itertools and collections a couple times and know when and where to use them.

Eela6 fucked around with this message at 21:57 on Feb 16, 2017

# ? Feb 16, 2017 21:46

Adbot: ADBOT LOVES YOU

# ? Apr 28, 2024 19:00

SelfOM: Jun 15, 2010

What is the best way to generate a sorted random int array of variable length in Cython (this array is generated in a loop, which I don' know how to type)? I'm trying to weave two arrays together, where switchover positions occur at the random indexes.

# ? Feb 17, 2017 22:55

SelfOM: Jun 15, 2010

Rosalind posted:

My questions now are: 1. How the heck does anyone ever learn how to use this? I'm the first to admit that I'm no programming guru, but I have some experience with several different languages and actually getting started with Python makes absolutely no sense. Everything feels like the biggest clusterfuck of dependencies and "install X, Y, and Z to get W to work but to get X to work install A and B which require C to run on Windows." 2. Is there a simple, idiot-proof (because that's what I am apparently) guide to moving from R to Python for data analysis and hopefully eventually machine learning?

I do a lot of R and python. For windows I would use anaconda: https://conda.io/miniconda.html It's not perfect, but I find myself using it for non-python libraries as well. I don't recommend rpy2 at all for beginners, its for importing R objects, which often times you can just use subprocess and grab the output data. For R equivalent tables, install pandas. Conda should handle the dependencies.

# ? Feb 17, 2017 23:11

Eela6: May 25, 2007; Shredded Hen

SelfOM posted:

What is the best way to generate a sorted random int array of variable length in Cython (this array is generated in a loop, which I don' know how to type)? I'm trying to weave two arrays together, where switchover positions occur at the random indexes.

Python code:

import numpy as np
def sorted_random_ints(low, high=None, size=None):
    random_ints = np.random.randint(low, high, size)
    return np.sort(random_ints)

np.sort defaults to quicksort, also has mergesort and heapsort. For integers in a restricted range, counting sort should be fastest of all - if you're really burning for speed, you can use that. But this should be pretty fast.

Eela6 fucked around with this message at 23:33 on Feb 17, 2017

# ? Feb 17, 2017 23:29

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

I wanna use virtualenv like all the cool kids but one of the python packages/libraries I'm using isn't managed by pip. It's something I need to compile myself with cmake and is linked against the systemwide Intel MKL and other dependencies. It doesn't show up when I type "pip list" but is importable/usable.

How can I use virtualenv --no-site-packages to get a clean environment, but with this one package included/available? Can I just symlink something to my virtualenv folder (and what)?

# ? Feb 19, 2017 12:37

QuarkJets: Sep 8, 2008

Boris Galerkin posted:

I wanna use virtualenv like all the cool kids but one of the python packages/libraries I'm using isn't managed by pip. It's something I need to compile myself with cmake and is linked against the systemwide Intel MKL and other dependencies. It doesn't show up when I type "pip list" but is importable/usable.

How can I use virtualenv --no-site-packages to get a clean environment, but with this one package included/available? Can I just symlink something to my virtualenv folder (and what)?

Your virtualenv has its own pip, distinct from the system pip; if you use your virtualenv's pip to install the package that you've compiled, then the package will only be installed to that virtualenv and nowhere else.

Installing the package depends on how it's setup; if it has a setup.py or a requirements.txt, etc. But basically you can use pip to install local packages as well as remote packages

# ? Feb 19, 2017 13:21

KernelSlanders: May 27, 2013; Rogue operating systems on occasion spread lies and rumors about me.

I don't disagree with anything QuarkJets said, but if your c dependency needs to be built against a different target you're probably going to want a full vm.

# ? Feb 19, 2017 17:18

pangstrom: Jan 25, 2003; Wedge Regret

Just as something that bothers me continuously (much like factors in R), is there a nice under-the-hood or mathematical reason why when subsetting lists or numpy arrays the last number is excluded? (e.g., mylist[1:3] means the 2nd and 3rd element, but not the 4th) I'm fine with zero-indexing for some reason but that one fucks me up on the regular.

# ? Feb 20, 2017 02:12

shrike82: Jun 11, 2005

Easy way to count the number of elements being referenced for one thing.

# ? Feb 20, 2017 02:15

QuarkJets: Sep 8, 2008

pangstrom posted:

Just as something that bothers me continuously (much like factors in R), is there a nice under-the-hood or mathematical reason why when subsetting lists or numpy arrays the last number is excluded? (e.g., mylist[1:3] means the 2nd and 3rd element, but not the 4th) I'm fine with zero-indexing for some reason but that one fucks me up on the regular.

I think it was just an arbitrary design choice, but it does make for some nice symmetries for things like half-open indices. Suppose you are splitting a string at 2 different points:

Python code:

x = 'this is a string'
i=3
j=7
a = x[:i]
b = x[i:j]
c = x[j:]

If the upper-bound was inclusive, then you'd have to do something more complicated with the indexing in the example above.

It also means that the length of a slice is equal to the difference of its indices; x[3:5] has a length of 2

# ? Feb 20, 2017 02:54

pangstrom: Jan 25, 2003; Wedge Regret

I guess that's as good a reason as I could have hoped for. Did/have other languages make/made that design choice?

# ? Feb 20, 2017 03:18

QuarkJets: Sep 8, 2008

pangstrom posted:

I guess that's as good a reason as I could have hoped for. Did/have other languages make/made that design choice?

I know that Go does it the Python way, but Fortran, Matlab, and Julia do it the other way, where the upper bound is inclusive.

There are advantages and disadvantages to both approaches, much like 1-based vs 0-based indexing

# ? Feb 20, 2017 03:26

Fergus Mac Roich: Nov 5, 2008; Soiled Meat

I feel that half-open has been fairly universal in the languages that I've used. For example, Java's Arrays.sort() uses half-open indexing. Dijkstra essentially makes an argument for it here

# ? Feb 20, 2017 03:29

pangstrom: Jan 25, 2003; Wedge Regret

Thanks all. Yeah I'm sure if it were the first thing I picked up I wouldn't be making this mistake so much (and I think I will do better going forward now that I have this conversation tagged to it).

# ? Feb 20, 2017 19:43

SelfOM: Jun 15, 2010

Eela6 posted:

np.sort defaults to quicksort, also has mergesort and heapsort. For integers in a restricted range, counting sort should be fastest of all - if you're really burning for speed, you can use that. But this should be pretty fast.

Thanks this is the solution I'm using at the moment.

# ? Feb 20, 2017 21:15

Eela6: May 25, 2007; Shredded Hen

SelfOM posted:

Thanks this is the solution I'm using at the moment.

Numpy's or counting sort? If you have a couple spare minutes, I'd be interested in you profiling the two and seeing what the actual speed difference is for your application.

# ? Feb 21, 2017 00:27

VikingofRock: Aug 24, 2008

Fergus Mac Roich posted:

I feel that half-open has been fairly universal in the languages that I've used. For example, Java's Arrays.sort() uses half-open indexing. Dijkstra essentially makes an argument for it here

Yeah the only exception to this in the languages I've used is in Haskell, where [1 .. 5] is the list [1, 2, 3, 4, 5]. Generally I like half open indices, and the only time it's bitten me is in languages where I can't easily express a max bound for some reason. For example in Rust, expressing the range (i .. 256u8) doesn't work because you can't have 256 as an unsigned 8-bit value. They've talked about adding an inclusive range syntax to rectify this, so I could do (i ... 255u8) instead.

# ? Feb 21, 2017 05:54

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

Anybody have any suggestions for my logging question?

# ? Feb 22, 2017 22:53

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

fletcher posted:

Anybody have any suggestions for my logging question?

Holy poo poo I think I finally fixed it!! Solution was to use a filter to filter log messages by thread id.

https://nelsonslog.wordpress.com/2015/01/14/python-logging-different-threads-to-different-files/

https://gist.github.com/migurski/e73faf00d7e1c5e44bd6

Updated my example repo with the fix: https://github.com/fletchowns/logging_test

Hooray!

# ? Feb 23, 2017 20:57

huhu: Feb 24, 2006

For anyone thinking about getting an Algorithms book for Python, don't buy this -
https://www.amazon.com/Data-Structu...gorithms+python

It boggles my mind that it managed to get 76 ratings of 4 stars. I'm catching at least one error every 5 to 10 pages and some of them are quite significant.

# ? Feb 24, 2017 23:20

Eela6: May 25, 2007; Shredded Hen

I gave a speech at San Diego Python's monthly meet-up last night. It went really well! Slides available at https://tinyurl.com/elements-of-style-I

# ? Feb 25, 2017 00:08

GobiasIndustries: Dec 14, 2007; Lipstick Apathy

Learning python, wrote a very simple script that asks for an input and prints said input. I'm on OSX, Sublime Text. When I go to Build, anything I type into the is just ignored. So I guess my question is, how do I test python scripts asking for raw_input?

# ? Feb 25, 2017 05:33

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

I've no experience with it but seems you can't do that in Sublime Text without a REPL plugin, all you have there is a basic output window

The easiest way is to open a terminal window or whatever in the folder you've saved your script, and just do python scriptname.py, and then you can interact with it. Different editors and IDEs will come with something like this built in, or you can install plugins like the Sublime one up there

# ? Feb 25, 2017 07:23

huhu: Feb 24, 2006

code:


import configparser

config = configparser.ConfigParser()
config.read('config.ini')

some_value = config['DEFAULT']['some_value']

some_function(some_value)

Is there a more elegant way to get values from a config file and then use them in a Python script or is this it?

# ? Feb 25, 2017 17:13

a witch: Jan 12, 2017

Well, what do you mean when you say "elegant"? What don't you like about the code you posted ?

# ? Feb 25, 2017 20:52

Nippashish: Nov 2, 2005; Let me see you dance!

The first problem is storing configuration in ini files. There's no good way to make non-hierarchical configuration not be annoying. I usually end up writing a dict-like class that lets me use thing.element instead of thing['element'] and writing all my configuration in config.py files.

# ? Feb 25, 2017 21:00

QuarkJets: Sep 8, 2008

I'm with Nippashish; ConfigParser feels clunky. If you just want to define a bunch of parameters that are used across a project, placing them in a config.py and importing them as-needed is elegant and simple. Wrapping them in a dict-like implementation and then importing that is good, too.

ConfigParser is designed for projects that want users who are familiar with Windows ini files and who are afraid of opening .py files to be able to modify configurable parameters, with built-in error checking. That's a pretty narrow scope. If your users fall out of that scope, or if you don't really have many or any users, then ConfigParser adds nothing on top of feeling lovely to use. The error-checking is nice if you're worried about users really loving things up and being unable to recover

# ? Feb 25, 2017 21:37

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

config.py ftw.

Also, like Nippashish, I'll use something to make it where I can use dot-notation rather than accessing keys because I just don't like accessing keys. Feels and looks ugly.

# ? Feb 25, 2017 22:21

huhu: Feb 24, 2006

Could you guys give me an example of what your settings.py files my look like? Also, does the dot notation you use autofill in something like Pycharm? I'd like to be able to type my_config.a and have all the settings with a pop up. Perhaps I'm going about this wrong.

Edit: I'm probing because the project I'm trying to work on might be useful for others. Probably not but I'd like the practice of making my stuff more user friendly.

Edit2: I think I can clarify a bit. I have a code like so:

code:

def dropbox_upload(file):
    dropbox_auth_token = open("dropbox_auth.txt", "r").read()
    #do all the uploading stuff here

Where I'll call it later on in several other different functions. I feel like the first line of the dropbox_upload function is bed. My two other thoughts are to do:

code:

import my_config

def dropbox_upload(file, auth_key):
    dropbox_auth_token = open(auth_key, "r").read()
    #do all the uploading stuff here

dropbox_upload(file, my_config.auth_key)

code:

import my_config

def dropbox_upload(file):
    dropbox_auth_token = open(my_config.auth_key, "r").read()
    #do all the uploading stuff here

But I think both of those are sloppy. How best could this be done?

huhu fucked around with this message at 01:58 on Feb 26, 2017

# ? Feb 26, 2017 00:39

QuarkJets: Sep 8, 2008

huhu posted:

Could you guys give me an example of what your settings.py files my look like? Also, does the dot notation you use autofill in something like Pycharm? I'd like to be able to type my_config.a and have all the settings with a pop up. Perhaps I'm going about this wrong.

Edit: I'm probing because the project I'm trying to work on might be useful for others. Probably not but I'd like the practice of making my stuff more user friendly.

Here's maybe the most straightforward example

Python code:

#settings.py
do_count_butts = True

Python code:

# some other file in the project
from settings import do_count_butts

#...

if do_count_butts:
    print 'it seems that we are counting butts now'

Pycharm will definitely autocomplete if you do it this way. It will also autocomplete if you just add properties to a class:

Python code:

#settings.py
class Params:
    do_count_butts = True

Python code:

# some other file
from settings import Params
if Params.do_count_butts:
  # etc

The world is your oyster

# ? Feb 26, 2017 02:02

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

huhu posted:

code:
import configparser

config = configparser.ConfigParser()
config.read('config.ini')

some_value = config['DEFAULT']['some_value']

some_function(some_value)
Is there a more elegant way to get values from a config file and then use them in a Python script or is this it?

You could think about using oslo.config, which is a configuration parser library used by OpenStack that is reasonably nice to use (I've used it in some projects). The downside is that it will drag in a number of other dependencies - not too many by OpenStack standards, but maybe more than you want to deal with: http://git.openstack.org/cgit/openstack/oslo.config/tree/requirements.txt

# ? Feb 26, 2017 16:58

Olly the Otter: Jul 22, 2007

Does anyone know how to get PyDev to recognize libraries that are just .so files?

For example, it keeps telling me that cjson and resource are unresolved imports, even though these are present and in standard locations:
/usr/lib/python2.7/dist-packages/cjson.x86_64-linux-gnu.so
/usr/lib/python2.7/lib-dynload/resource.x86_64-linux-gnu.so

It also flags as "undefined variable" any attempt to use these libraries, for example by calling cjson.decode.

Both of these folders (/usr/lib/python2.7/dist-packages and /usr/lib/python2.7/lib-dynload) are listed in the system PYTHONPATH under Python Interpreters in the settings dialog. Googling this problem yields numerous examples where folders weren't in that list, but that doesn't appear to be my problem. I'm guessing it has more to do with that it's a .so file rather than .py, or that the filename is decorated with architecture details. Of course, my program has no difficulty with finding and using these libraries at runtime.

I know that I can decorate the code with UnresolvedImport etc., but I'd rather figure out what's wrong and fix it.

# ? Feb 27, 2017 04:57

Dominoes: Sep 20, 2007

Hey dudes. What are the best ways to learn Python, for someone with no experience? I learned from Codeacademy, which was OK. The OP looks out of date.

Dominoes fucked around with this message at 17:16 on Feb 27, 2017

# ? Feb 27, 2017 17:05

Tigren: Oct 3, 2003

Dominoes posted:

Hey dudes. What are the best ways to learn Python, for someone with no experience? I learned from Codeacademy, which was OK. The OP looks out of date.

https://learnpythonthehardway.org/python3/

# ? Feb 27, 2017 18:54

Dominoes: Sep 20, 2007

Isn't that by the guy who ranted about Python 3 in the old V, then justified it recently with odd reasoning?

Dominoes fucked around with this message at 19:13 on Feb 27, 2017

# ? Feb 27, 2017 19:09

VikingofRock: Aug 24, 2008

Dominoes posted:

Hey dudes. What are the best ways to learn Python, for someone with no experience? I learned from Codeacademy, which was OK. The OP looks out of date.

I've heard a lot of good things about Fluent Python.

# ? Feb 27, 2017 19:13

Eela6: May 25, 2007; Shredded Hen

Tigren posted:

https://learnpythonthehardway.org/python3/

I do not care for this book. The author seems more interested in showing that he is smart than teaching the audience how to write code. He's got a 'first you must sweep the floor before you can learn how to fight' approach to teaching. Python is not Karate.

I have heard good things about this course

When you say you are an absolute beginner, do you mean to programming, or just to python?

Fluent Python is the best programming book I know of but it is not appropriate for beginners. It's a book for the intermediate python programmer. It might be fun to get anyways, as something to page through and look forward to as you learn the basics somewhere else.

I am of the opinion that the best way to learn how to program is by doing so. Pick a project, any project, and try it in python! If you're of a mathematical bent, you might like Project Euler

The best way to get Python running on your computer with all the libraries and additions you might want is Anaconda . Download it! You probably want the version with Python 3.6. The built-in IDE, spyder, is not as full-featured as some others but is very easy to use; I often use it myself. If you aren't using an IDE yet I would try using that or PyCharm!

There are tons of tutorials from python experts archived on youtube. If you are confused or want to explore a specific subject, try looking it up, plus the term 'PyCon', on youtube. You will probably find a whole talk on that subject!

Also, if you have any questions, you can always ask here. The beginning is always the hardest. Stick with it and study hard, and you will feel like a pro in no time.

Dominoes posted:

Isn't that by the guy who ranted about Python 3 in the old V, then justified it recently with odd reasoning?

Yes he did. He now has an amazing 'puppetmaster' defense of the old parts where everyone made fun of him, which I won't bother to link.

Eela6 fucked around with this message at 19:35 on Feb 27, 2017

# ? Feb 27, 2017 19:20

Dominoes: Sep 20, 2007

Thanks for the Udacity rec. The guy looking for the rec is new to programming, other than a C++ course in college a decade ago.

# ? Feb 27, 2017 19:28

onionradish: Jul 6, 2006; That's spicy.

Dominoes posted:

Isn't that by the guy who ranted about Python 3 in the old V, then doubled down recently?

It is; I'd avoid him on principle based on how he lashed out against critics, aside from suspecting anything he says as being out of date since he's been so anti-Python3. Suddenly, he'd be worth listening to?

Automate the Boring Stuff is probably the most-recommended starter since it emphasizes doing something practical with the language.

Other sources I found useful when learning:

PyCharm as an IDE to kick me in the pants on style and provide easy access to method calls
Reddit's /learnpython wiki
Reddit's /learnpython and /Python groups; /learnpython has a mix of beginner questions and lazy student questions, which means anyone can ask beginner questions without standing out, or see how many of the existing questions you already know the answer to - it's something a beginner outgrows, which is a decent milestone
PyVideo.org, which has hundreds of video sessions from Python conferences, ranging from brief overviews of modules, to 3+ hour tutorials on topics from basic Python, to good Pythonic code (Hettinger, et al), to heavily-specific scientific topics (using modules for DNA analysis, real-time bidding for online ads, etc.)
DevDocs.io to copy a local, offline, browser-accessible version of Python docs (as well as bunch of other languages)

# ? Feb 27, 2017 19:33

Adbot: ADBOT LOVES YOU

# ? Apr 28, 2024 19:00

Eela6: May 25, 2007; Shredded Hen

BTW, should we make a new Python thread? This one is pretty crusty. It might be nice to have an updated OP with some of this material.

# ? Feb 27, 2017 19:41

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »