[quote="huhu"] Uneducated question - why does memory management matter? I understand why it would with something like an Arduino which has almost no memory but why would that matter on something like a computer with so much more memory? For context, I'm reading about arrays vs linked lists. [\quote] Speed, scaling, and professionalism. Speed: Proper memory use is faster. If you manage your memory well, your CPU doesn't have to constantly load things in and out of the various caches. In many cases, proper memory management can lewd to 20-100x speedup of your code. If you're so wasteful of memory that you start having to constantly load things in and out of the drive, this is another couple orders of magnitude slower. Scaling: Say you write a text-processing script of some kind that has to iterate over your material multiple times. So to make things easier, you turn it into a list, you load the whole thing into memory, then check the text. Most text files are under a megabyte, so who cares, right? This world fine for 99% of text corpuses. Then you get a 120gb text corpus - say, every single SA forum post. It's 50gb. You run your script, it locks your computer, you look silly. Professionalism: You're a programmer. You should write good code, and take pride in it. You don't have to optimize every single thing, but if you always take that extra bit of time to make your code clean and efficient, it will make you a better coder. If you always do it right, it will become a matter of habit, and writing good code when it matters will become trivial. You want your default code to be good code. Memory management is only part of this, but people who are lazy and sloppy with memory are sloppy elsewhere. The buck stops with you. Luckily, good memory use is easy in python. Use generator comprehensions rather than list comprehensions. On that note, don't return a list if you can yield the contents. Use the correct data structures. Write small functions, so that intermediate variables exit scope and are deleted by the GC. Read through itertools and collections a couple times and know when and where to use them. Eela6 fucked around with this message at 21:57 on Feb 16, 2017 |
|
# ? Feb 16, 2017 21:46 |
|
|
# ? May 8, 2024 05:56 |
|
What is the best way to generate a sorted random int array of variable length in Cython (this array is generated in a loop, which I don' know how to type)? I'm trying to weave two arrays together, where switchover positions occur at the random indexes.
|
# ? Feb 17, 2017 22:55 |
|
Rosalind posted:My questions now are: 1. How the heck does anyone ever learn how to use this? I'm the first to admit that I'm no programming guru, but I have some experience with several different languages and actually getting started with Python makes absolutely no sense. Everything feels like the biggest clusterfuck of dependencies and "install X, Y, and Z to get W to work but to get X to work install A and B which require C to run on Windows." 2. Is there a simple, idiot-proof (because that's what I am apparently) guide to moving from R to Python for data analysis and hopefully eventually machine learning? I do a lot of R and python. For windows I would use anaconda: https://conda.io/miniconda.html It's not perfect, but I find myself using it for non-python libraries as well. I don't recommend rpy2 at all for beginners, its for importing R objects, which often times you can just use subprocess and grab the output data. For R equivalent tables, install pandas. Conda should handle the dependencies.
|
# ? Feb 17, 2017 23:11 |
SelfOM posted:What is the best way to generate a sorted random int array of variable length in Cython (this array is generated in a loop, which I don' know how to type)? I'm trying to weave two arrays together, where switchover positions occur at the random indexes. Python code:
Eela6 fucked around with this message at 23:33 on Feb 17, 2017 |
|
# ? Feb 17, 2017 23:29 |
|
I wanna use virtualenv like all the cool kids but one of the python packages/libraries I'm using isn't managed by pip. It's something I need to compile myself with cmake and is linked against the systemwide Intel MKL and other dependencies. It doesn't show up when I type "pip list" but is importable/usable. How can I use virtualenv --no-site-packages to get a clean environment, but with this one package included/available? Can I just symlink something to my virtualenv folder (and what)?
|
# ? Feb 19, 2017 12:37 |
|
Boris Galerkin posted:I wanna use virtualenv like all the cool kids but one of the python packages/libraries I'm using isn't managed by pip. It's something I need to compile myself with cmake and is linked against the systemwide Intel MKL and other dependencies. It doesn't show up when I type "pip list" but is importable/usable. Your virtualenv has its own pip, distinct from the system pip; if you use your virtualenv's pip to install the package that you've compiled, then the package will only be installed to that virtualenv and nowhere else. Installing the package depends on how it's setup; if it has a setup.py or a requirements.txt, etc. But basically you can use pip to install local packages as well as remote packages
|
# ? Feb 19, 2017 13:21 |
|
I don't disagree with anything QuarkJets said, but if your c dependency needs to be built against a different target you're probably going to want a full vm.
|
# ? Feb 19, 2017 17:18 |
|
Just as something that bothers me continuously (much like factors in R), is there a nice under-the-hood or mathematical reason why when subsetting lists or numpy arrays the last number is excluded? (e.g., mylist[1:3] means the 2nd and 3rd element, but not the 4th) I'm fine with zero-indexing for some reason but that one fucks me up on the regular.
|
# ? Feb 20, 2017 02:12 |
|
Easy way to count the number of elements being referenced for one thing.
|
# ? Feb 20, 2017 02:15 |
|
pangstrom posted:Just as something that bothers me continuously (much like factors in R), is there a nice under-the-hood or mathematical reason why when subsetting lists or numpy arrays the last number is excluded? (e.g., mylist[1:3] means the 2nd and 3rd element, but not the 4th) I'm fine with zero-indexing for some reason but that one fucks me up on the regular. I think it was just an arbitrary design choice, but it does make for some nice symmetries for things like half-open indices. Suppose you are splitting a string at 2 different points: Python code:
It also means that the length of a slice is equal to the difference of its indices; x[3:5] has a length of 2
|
# ? Feb 20, 2017 02:54 |
|
I guess that's as good a reason as I could have hoped for. Did/have other languages make/made that design choice?
|
# ? Feb 20, 2017 03:18 |
|
pangstrom posted:I guess that's as good a reason as I could have hoped for. Did/have other languages make/made that design choice? I know that Go does it the Python way, but Fortran, Matlab, and Julia do it the other way, where the upper bound is inclusive. There are advantages and disadvantages to both approaches, much like 1-based vs 0-based indexing
|
# ? Feb 20, 2017 03:26 |
|
I feel that half-open has been fairly universal in the languages that I've used. For example, Java's Arrays.sort() uses half-open indexing. Dijkstra essentially makes an argument for it here
|
# ? Feb 20, 2017 03:29 |
|
Thanks all. Yeah I'm sure if it were the first thing I picked up I wouldn't be making this mistake so much (and I think I will do better going forward now that I have this conversation tagged to it).
|
# ? Feb 20, 2017 19:43 |
|
Eela6 posted:np.sort defaults to quicksort, also has mergesort and heapsort. For integers in a restricted range, counting sort should be fastest of all - if you're really burning for speed, you can use that. But this should be pretty fast. Thanks this is the solution I'm using at the moment.
|
# ? Feb 20, 2017 21:15 |
SelfOM posted:Thanks this is the solution I'm using at the moment. Numpy's or counting sort? If you have a couple spare minutes, I'd be interested in you profiling the two and seeing what the actual speed difference is for your application.
|
|
# ? Feb 21, 2017 00:27 |
Fergus Mac Roich posted:I feel that half-open has been fairly universal in the languages that I've used. For example, Java's Arrays.sort() uses half-open indexing. Dijkstra essentially makes an argument for it here Yeah the only exception to this in the languages I've used is in Haskell, where [1 .. 5] is the list [1, 2, 3, 4, 5]. Generally I like half open indices, and the only time it's bitten me is in languages where I can't easily express a max bound for some reason. For example in Rust, expressing the range (i .. 256u8) doesn't work because you can't have 256 as an unsigned 8-bit value. They've talked about adding an inclusive range syntax to rectify this, so I could do (i ... 255u8) instead.
|
|
# ? Feb 21, 2017 05:54 |
Anybody have any suggestions for my logging question?
|
|
# ? Feb 22, 2017 22:53 |
fletcher posted:Anybody have any suggestions for my logging question? Holy poo poo I think I finally fixed it!! Solution was to use a filter to filter log messages by thread id. https://nelsonslog.wordpress.com/2015/01/14/python-logging-different-threads-to-different-files/ https://gist.github.com/migurski/e73faf00d7e1c5e44bd6 Updated my example repo with the fix: https://github.com/fletchowns/logging_test Hooray!
|
|
# ? Feb 23, 2017 20:57 |
|
For anyone thinking about getting an Algorithms book for Python, don't buy this - https://www.amazon.com/Data-Structu...gorithms+python It boggles my mind that it managed to get 76 ratings of 4 stars. I'm catching at least one error every 5 to 10 pages and some of them are quite significant.
|
# ? Feb 24, 2017 23:20 |
I gave a speech at San Diego Python's monthly meet-up last night. It went really well! Slides available at https://tinyurl.com/elements-of-style-I
|
|
# ? Feb 25, 2017 00:08 |
|
Learning python, wrote a very simple script that asks for an input and prints said input. I'm on OSX, Sublime Text. When I go to Build, anything I type into the is just ignored. So I guess my question is, how do I test python scripts asking for raw_input?
|
# ? Feb 25, 2017 05:33 |
|
I've no experience with it but seems you can't do that in Sublime Text without a REPL plugin, all you have there is a basic output window The easiest way is to open a terminal window or whatever in the folder you've saved your script, and just do python scriptname.py, and then you can interact with it. Different editors and IDEs will come with something like this built in, or you can install plugins like the Sublime one up there
|
# ? Feb 25, 2017 07:23 |
|
code:
Is there a more elegant way to get values from a config file and then use them in a Python script or is this it?
|
# ? Feb 25, 2017 17:13 |
|
Well, what do you mean when you say "elegant"? What don't you like about the code you posted ?
|
# ? Feb 25, 2017 20:52 |
|
The first problem is storing configuration in ini files. There's no good way to make non-hierarchical configuration not be annoying. I usually end up writing a dict-like class that lets me use thing.element instead of thing['element'] and writing all my configuration in config.py files.
|
# ? Feb 25, 2017 21:00 |
|
I'm with Nippashish; ConfigParser feels clunky. If you just want to define a bunch of parameters that are used across a project, placing them in a config.py and importing them as-needed is elegant and simple. Wrapping them in a dict-like implementation and then importing that is good, too. ConfigParser is designed for projects that want users who are familiar with Windows ini files and who are afraid of opening .py files to be able to modify configurable parameters, with built-in error checking. That's a pretty narrow scope. If your users fall out of that scope, or if you don't really have many or any users, then ConfigParser adds nothing on top of feeling lovely to use. The error-checking is nice if you're worried about users really loving things up and being unable to recover
|
# ? Feb 25, 2017 21:37 |
|
config.py ftw. Also, like Nippashish, I'll use something to make it where I can use dot-notation rather than accessing keys because I just don't like accessing keys. Feels and looks ugly.
|
# ? Feb 25, 2017 22:21 |
|
Could you guys give me an example of what your settings.py files my look like? Also, does the dot notation you use autofill in something like Pycharm? I'd like to be able to type my_config.a and have all the settings with a pop up. Perhaps I'm going about this wrong. Edit: I'm probing because the project I'm trying to work on might be useful for others. Probably not but I'd like the practice of making my stuff more user friendly. Edit2: I think I can clarify a bit. I have a code like so: code:
code:
code:
huhu fucked around with this message at 01:58 on Feb 26, 2017 |
# ? Feb 26, 2017 00:39 |
|
huhu posted:Could you guys give me an example of what your settings.py files my look like? Also, does the dot notation you use autofill in something like Pycharm? I'd like to be able to type my_config.a and have all the settings with a pop up. Perhaps I'm going about this wrong. Here's maybe the most straightforward example Python code:
Python code:
Python code:
Python code:
|
# ? Feb 26, 2017 02:02 |
|
huhu posted:
You could think about using oslo.config, which is a configuration parser library used by OpenStack that is reasonably nice to use (I've used it in some projects). The downside is that it will drag in a number of other dependencies - not too many by OpenStack standards, but maybe more than you want to deal with: http://git.openstack.org/cgit/openstack/oslo.config/tree/requirements.txt
|
# ? Feb 26, 2017 16:58 |
|
Does anyone know how to get PyDev to recognize libraries that are just .so files? For example, it keeps telling me that cjson and resource are unresolved imports, even though these are present and in standard locations: /usr/lib/python2.7/dist-packages/cjson.x86_64-linux-gnu.so /usr/lib/python2.7/lib-dynload/resource.x86_64-linux-gnu.so It also flags as "undefined variable" any attempt to use these libraries, for example by calling cjson.decode. Both of these folders (/usr/lib/python2.7/dist-packages and /usr/lib/python2.7/lib-dynload) are listed in the system PYTHONPATH under Python Interpreters in the settings dialog. Googling this problem yields numerous examples where folders weren't in that list, but that doesn't appear to be my problem. I'm guessing it has more to do with that it's a .so file rather than .py, or that the filename is decorated with architecture details. Of course, my program has no difficulty with finding and using these libraries at runtime. I know that I can decorate the code with UnresolvedImport etc., but I'd rather figure out what's wrong and fix it.
|
# ? Feb 27, 2017 04:57 |
|
Hey dudes. What are the best ways to learn Python, for someone with no experience? I learned from Codeacademy, which was OK. The OP looks out of date.
Dominoes fucked around with this message at 17:16 on Feb 27, 2017 |
# ? Feb 27, 2017 17:05 |
|
Dominoes posted:Hey dudes. What are the best ways to learn Python, for someone with no experience? I learned from Codeacademy, which was OK. The OP looks out of date. https://learnpythonthehardway.org/python3/
|
# ? Feb 27, 2017 18:54 |
|
Isn't that by the guy who ranted about Python 3 in the old V, then justified it recently with odd reasoning?
Dominoes fucked around with this message at 19:13 on Feb 27, 2017 |
# ? Feb 27, 2017 19:09 |
Dominoes posted:Hey dudes. What are the best ways to learn Python, for someone with no experience? I learned from Codeacademy, which was OK. The OP looks out of date. I've heard a lot of good things about Fluent Python.
|
|
# ? Feb 27, 2017 19:13 |
I do not care for this book. The author seems more interested in showing that he is smart than teaching the audience how to write code. He's got a 'first you must sweep the floor before you can learn how to fight' approach to teaching. Python is not Karate. I have heard good things about this course When you say you are an absolute beginner, do you mean to programming, or just to python? Fluent Python is the best programming book I know of but it is not appropriate for beginners. It's a book for the intermediate python programmer. It might be fun to get anyways, as something to page through and look forward to as you learn the basics somewhere else. I am of the opinion that the best way to learn how to program is by doing so. Pick a project, any project, and try it in python! If you're of a mathematical bent, you might like Project Euler The best way to get Python running on your computer with all the libraries and additions you might want is Anaconda . Download it! You probably want the version with Python 3.6. The built-in IDE, spyder, is not as full-featured as some others but is very easy to use; I often use it myself. If you aren't using an IDE yet I would try using that or PyCharm! There are tons of tutorials from python experts archived on youtube. If you are confused or want to explore a specific subject, try looking it up, plus the term 'PyCon', on youtube. You will probably find a whole talk on that subject! Also, if you have any questions, you can always ask here. The beginning is always the hardest. Stick with it and study hard, and you will feel like a pro in no time. Dominoes posted:Isn't that by the guy who ranted about Python 3 in the old V, then justified it recently with odd reasoning? Yes he did. He now has an amazing 'puppetmaster' defense of the old parts where everyone made fun of him, which I won't bother to link. Eela6 fucked around with this message at 19:35 on Feb 27, 2017 |
|
# ? Feb 27, 2017 19:20 |
|
Thanks for the Udacity rec. The guy looking for the rec is new to programming, other than a C++ course in college a decade ago.
|
# ? Feb 27, 2017 19:28 |
|
Dominoes posted:Isn't that by the guy who ranted about Python 3 in the old V, then doubled down recently? Automate the Boring Stuff is probably the most-recommended starter since it emphasizes doing something practical with the language. Other sources I found useful when learning:
|
# ? Feb 27, 2017 19:33 |
|
|
# ? May 8, 2024 05:56 |
BTW, should we make a new Python thread? This one is pretty crusty. It might be nice to have an updated OP with some of this material.
|
|
# ? Feb 27, 2017 19:41 |