|
supercrooky posted:Anyone have any suggestions for a memory efficient 2D array of ints with reasonable indexing speed? I've gone down the list of lists route (too much memory usage) and the numpy array route (slow indexing). My next thought is wrapping the builtin array object to have a 2D like interface.
|
# ? Aug 25, 2014 19:50 |
|
|
# ? May 8, 2024 08:25 |
|
supercrooky posted:Anyone have any suggestions for a memory efficient 2D array of ints with reasonable indexing speed? I've gone down the list of lists route (too much memory usage) and the numpy array route (slow indexing). My next thought is wrapping the builtin array object to have a 2D like interface. What's the actual problem that has led you to needing a 2D array with faster indexing?
|
# ? Aug 25, 2014 19:59 |
|
QuarkJets posted:What's the actual problem that has led you to needing a 2D array with faster indexing? Needleman-Wunsch/Smith-Waterman algorithms for protein alignment. List of arrays is getting me 90% less memory usage for 10% additional runtime, so that's going to be good enough for now. Haven't tried numba yet.
|
# ? Aug 25, 2014 20:38 |
|
Is there something wrong withPython code:
|
# ? Aug 25, 2014 20:41 |
|
The last time I implemented a dynamic programming algorithm in Python and cared about speed my fastest iteration involved numpy typed memoryviews and Cython.
|
# ? Aug 25, 2014 20:43 |
|
The only time I ever used Smith-Waterman my sequences were too short to get any benefit out of using numpy over a mostly naive implementation :\ Looking at it again, I could probably abuse a multi-level generator to speed it up some
|
# ? Aug 25, 2014 22:01 |
|
I've used dicts with tuple keys before.Python code:
|
# ? Aug 26, 2014 20:40 |
|
Crosscontaminant posted:I've used dicts with tuple keys before. Dicts are usually not the answer to a question involving the words "memory-efficient".
|
# ? Aug 26, 2014 21:49 |
|
OnceIWasAnOstrich posted:Dicts are usually not the answer to a question involving the words "memory-efficient". If the choice is between a 1000x1000 dense array with 100 non-zero values, or a cheap-o sparse array using a dict with 100 items, the dict wins.
|
# ? Aug 26, 2014 22:43 |
|
I suppose that was so broad a statement as to be wrong. Dicts will be memory efficient for things hash tables are good for, although not especially so compared to other implementations you can use from libraries. The constantly doubling memory use usually bites me when I care about memory. In any case, for most dynamic programming algorithms including Smith-Waterman, the arrays won't be especially sparse.
OnceIWasAnOstrich fucked around with this message at 23:35 on Aug 26, 2014 |
# ? Aug 26, 2014 23:06 |
|
OnceIWasAnOstrich posted:I suppose that was so broad a statement as to be wrong. Dicts will be memory efficient for things hash tables are good for, although not especially so compared to other implementations you can use from libraries. The constantly doubling memory use usually bites me when I care about memory. In any case, for most dynamic programming algorithms including Smith-Waterman, the arrays won't be especially sparse. They start out quadrupling, actually.
|
# ? Aug 27, 2014 03:36 |
|
This is all somewhat babby's first python, but can someone tell me why code:
I'm pretty sure I indented it all correctly.
|
# ? Aug 27, 2014 18:35 |
|
EricBauman posted:This is all somewhat babby's first python, but can someone tell me why The print statement is outside of the for loop, so it will only execute after every url in "listoflinks.txt" is done being processed. Indent the print line statement one more time if you want it to print after each url is processed.
|
# ? Aug 27, 2014 18:40 |
|
vikingstrike posted:The print statement is outside of the for loop, so it will only execute after every url in "listoflinks.txt" is done being processed. Indent the print line statement one more time if you want it to print after each url is processed. Hmm, I think i misinterpreted and misplanned something in my planning for this. I wrote this a couple of weeks ago, then forgot about it because it didn't do what I wanted it to do. Now, a couple of minutes later, I've found out that it also doesn't stop at zero. It happily continues into negative numbers. I'll try to put some break things and other stuff like that in there.
|
# ? Aug 27, 2014 19:06 |
|
EricBauman posted:Now, a couple of minutes later, I've found out that it also doesn't stop at zero. It happily continues into negative numbers. I'll try to put some break things and other stuff like that in there. The for loop is where the subtraction happens, and the while loop is where it checks to see if your number is greater than 0. However, the while loop can't do that check until you've exited the for loop, and it's possible that your code inside the for loop continues to subtract into the negative numbers. You need some kind of check (like an if-statement) inside your for loop that prevents subtraction from happening if condition == 1 (and probably also breaks the loop). edit: Do you even need a while-loop? How many times are you going to loop through that list of urls anyway? If only once, you don't need the while. FoiledAgain fucked around with this message at 19:35 on Aug 27, 2014 |
# ? Aug 27, 2014 19:28 |
|
FoiledAgain posted:edit: Do you even need a while-loop? How many times are you going to loop through that list of urls anyway? If only once, you don't need the while. Yeah, I was about to post this. If you just need to run through the list once, then you just need the for loop there.
|
# ? Aug 27, 2014 19:39 |
|
EricBauman posted:Hmm, I think i misinterpreted and misplanned something in my planning for this. That's because the while loop is outside of the for loop. So the for loop will continue iterating over the listoflinks.txt, pausing 3 seconds between every line, until the entire listoflinks.txt has been read. After it's done iterating over the entire list, the while loop gets checked. So if you had 300 entries in listoflinks.txt, you'd wind up with a countdown of -100 before the while loop ever checks to see if countdown is less than 0. You also indented 4 spaces for one block and 7 spaces for another, you shouldn't indent by an odd number of spaces If you want to stop iterating after countdown becomes negative, then you can do this: Python code:
|
# ? Aug 27, 2014 19:48 |
|
EricBauman posted:Hmm, I think i misinterpreted and misplanned something in my planning for this.
|
# ? Aug 27, 2014 20:51 |
|
ShadowHawk posted:One possible source of your confusion is that while loops only check their condition when you get to the end of the loop (or start it for the first time) -- they don't continuously monitor the truthiness of their condition. Yeah, I think that's exactly what I was wrong about. This whole project is so a colleague can just more or less automatically download the last X files from a list that automatically logs changed files. The sleep is to prevent the process from clogging up our unstable wireless. I could do it without the countdown and the while loop, but then I'd need to change the list to only contain X items, and since the list is also used by other people for other purposes (I imagined my colleague could copy it to the folder the files have to go in), that's not really a possibility. Always fun, being the only non-techy team leader in an IT company and having to hack poo poo like this together together. Sure, I could ask a real programmer, but where's the fun in that?
|
# ? Aug 27, 2014 21:29 |
|
EricBauman posted:Yeah, I think that's exactly what I was wrong about. Is listoffiles.txt being appended to or prepended to? Because if it's being appended to you're not getting the last 200 files, you're getting the first 200 files. You should also check if a file has already been downloaded before you download it. And if you work with programmers isn't there one of them who would be willing to help you out instead of just writing it for you?
|
# ? Aug 28, 2014 00:39 |
|
Fixed the problem with the Windows COM ports, still not entirely sure what caused it.
dirtycajun fucked around with this message at 15:39 on Aug 28, 2014 |
# ? Aug 28, 2014 05:08 |
|
See above.
dirtycajun fucked around with this message at 15:39 on Aug 28, 2014 |
# ? Aug 28, 2014 05:26 |
|
Dren posted:Is listoffiles.txt being appended to or prepended to? It's being prepended to, fortunately. As to why I can't use programmers? Office politics, deadlines, stuff like that. I also want to learn to at least do some of this stuff myself, and I was working from home yesterday, looking at old projects that kind of dropped off my to do list.
|
# ? Aug 28, 2014 09:10 |
|
dirtycajun posted:Fixed the problem with the Windows COM ports, still not entirely sure what caused it. Not trying to be snarky. If you're not entirely sure what caused it, are you entirely sure of the solution?
|
# ? Aug 28, 2014 20:28 |
|
I'm just starting to learn Python to use in Maya and other CG programs to automate rigging and stuff like that. Python has started to make sense to me, and I'm learning the syntax okay, although a lot of the more abstract aspects of object oriented programming are still beyond my conceptual grasp. One thing I don't get about python is why this works: Python code:
|
# ? Aug 29, 2014 04:03 |
|
Ccs posted:I'm just starting to learn Python to use in Maya and other CG programs to automate rigging and stuff like that. Python has started to make sense to me, and I'm learning the syntax okay, although a lot of the more abstract aspects of object oriented programming are still beyond my conceptual grasp. This is called recursion and isn't specific to python.
|
# ? Aug 29, 2014 04:35 |
|
Anyone have success with bs4 and multiprocessing?Python code:
If I break out lxml will I run into the same problem? Changing the bs4 backend to lxml still produces errors. The bottle neck in the actual project are the handful of calls to the BeautifulSoup constructor that cost almost a second each on my machine. Check out this link, it answers your question very specifically. salisbury shake fucked around with this message at 18:04 on Aug 29, 2014 |
# ? Aug 29, 2014 04:57 |
|
Ccs posted:I'm just starting to learn Python to use in Maya and other CG programs to automate rigging and stuff like that. Python has started to make sense to me, and I'm learning the syntax okay, although a lot of the more abstract aspects of object oriented programming are still beyond my conceptual grasp. Here is a somewhat simpler explanation of recursion from the Think Python book: http://www.greenteapress.com/thinkpython/html/thinkpython006.html#toc59
|
# ? Aug 29, 2014 12:45 |
|
Wouldn't that function cause problems for any non-natural number?
|
# ? Aug 29, 2014 13:05 |
|
BabyFur Denny posted:Wouldn't that function cause problems for any non-natural number? To say nothing of strings.
|
# ? Aug 29, 2014 14:06 |
|
I have a jinja2 template that generates a page which displays a list of fields and their values based on the fields in a jinja2 context object. For example, it might show the key "unique_id" and "12345" as the value Instead of using this label, I need to label the field values based on a mapping contained in a separate API. The API returns a JSON object that shows the mapping from the key to the plain language label. For example, instead of displaying the key as "unique_id", it would be displayed as "Internal Unique Identifier". There are about a hundred of these fields, and the non-technical people that need to modify the mapping at the other side of the API will be making regular changes to this, so I cannot hardcode the plain language field names. What is the best way to make an API call and to pass the response object to a jinja2 template? EAT THE EGGS RICOLA fucked around with this message at 16:20 on Aug 29, 2014 |
# ? Aug 29, 2014 16:14 |
|
Ok, so that sort of data mashing is not the sort of thing you want to be doing inside of your template (down that path lies php). Get your data into shape, then pass it into the template. In your case, you're basically mashing together two dicts, so here's an outline of what your code code might look like.Python code:
Haystack fucked around with this message at 16:48 on Aug 29, 2014 |
# ? Aug 29, 2014 16:45 |
|
Haystack posted:Ok, so that sort of data mashing is not the sort of thing you want to be doing inside of your template (down that path lies php). Get your data into shape, then pass it into the template. In your case, you're basically mashing together two dicts, so here's an outline of what your code code might look like. I was afraid of that. This is a monstrosity of an application that someone has been doing crazy things with for years, and I was hoping to avoid delving into the guts straightaway. Thanks!
|
# ? Aug 29, 2014 16:50 |
|
I'm having some weird issues with sys.path... when creating new conda environments, it's adding ~/.local/lib to the path (which is where pip install --user puts stuff). I was not able to replicate this on another machine. Very confused. How do I even go about troubleshooting this? E: wait it is happening on my other machine. Maybe a conda bug. SurgicalOntologist fucked around with this message at 01:46 on Aug 30, 2014 |
# ? Aug 29, 2014 21:03 |
|
Conda issue discussed here. I'm trying to figure out how to fix it (one way is to always use the -s command-line flag, but that's annoying). The Python docs say: quote:site.ENABLE_USER_SITE I'm interested in the phrase or by an administator. I'm an administrator (on my own machines), how do I disable it for a particular interpreter? Any ideas? (The environment variable is a non-starter because I need to disable it only for specific interpreters) E: A second option, I suppose, is to stop using pip install --user and go back to sudo pip install for those unfortunate times when I need to install something system-wide. If the user-site directory doesn't exist, there's no problem. However, I don't like this because I make extensive use of the ~/.local prefix in general. SurgicalOntologist fucked around with this message at 03:18 on Aug 30, 2014 |
# ? Aug 30, 2014 03:15 |
|
Alright here's another question from me, the newbie Python learner. In this code:code:
When you define the variable "letter" in "word", how does "letter" know that it's supposed to represent the key to the dictionary? Why doesn't it assign itself to the value of the key? Or the key and value together? How is python this smart? Ccs fucked around with this message at 17:06 on Aug 31, 2014 |
# ? Aug 31, 2014 17:02 |
|
score[letter] takes the value of letter (which due to the line for letter in word, will be one letter of the word at a time) and uses that letter to look up a value in the dictionary score. For example, score['a'] returns 1. If word starts with a, then on the first iteration of the loop, letter will be 'a', and score[letter] will return 1. letter gets its values from word without reference to the dictionary score, if that's your confusion. If you try this with a word like 'abc123' you'll get an error when it tries to look up score['1']. In other words, Python isn't that smart, the code only works as long as the letters in word are keys in score. It's up to whoever wrote and uses this code to ensure that the letters are in the dictionary. E: And because I can't resist, allow me to improve the code: Python code:
SurgicalOntologist fucked around with this message at 17:51 on Aug 31, 2014 |
# ? Aug 31, 2014 17:43 |
|
As long as we are "improving":code:
|
# ? Sep 1, 2014 06:53 |
|
It's a cool idea and I know you're just trying things out but I like SurgicalOntologist's version better. It's a little faster and more straightforward.
|
# ? Sep 1, 2014 07:21 |
|
|
# ? May 8, 2024 08:25 |
|
Symbolic Butt posted:It's a cool idea and I know you're just trying things out but I like SurgicalOntologist's version better. It's a little faster and more straightforward. Oh, I do too, hence the scare quotes. Let's go another direction and make the ultra-flexible enterprise scorer that can perform arbitrary accumulations and transforms on the input with configurable decoders: code:
Edit: Unicode support is an open issue; see the GH tracker. BigRedDot fucked around with this message at 16:10 on Sep 1, 2014 |
# ? Sep 1, 2014 15:22 |