|
Cingulate posted:Hm. In that case, I'm going to make my suggestion to Dominoes more clear: if solving linear systems is a limiting factor in your stuff, maybe take a look at MATLAB's mldivide, which
|
# ? Mar 28, 2015 13:49 |
|
|
# ? May 9, 2024 07:52 |
|
Dominoes posted:Specifically, I'm running a few operations on millions of dataset combinations. I'm able to optimize most of them well with Numba, or they're quick enough not to be an issue. The holdup is using a linear regression (specifically statsmodels' GLM at the moment) to find residuals. It runs slow compared to the rest of my calculations. - check if there is a special property of the matrices you can exploit - are they e.g. sparse, or square? - are you using the best algorithm for solving that kind of problem? I assume statsmodels calls scipy or numpy for its linear systems, and IIRC neither ships UMFPACK - do you repeatedly solve y = B*x for the same x? In that case, you can store and recycle the factorisation for massive speed boosts For me, the optimal solution turned out to actually be making everything sparse and calling MATLAB's "\" once for all my observation matrices sharing the same predictor matrix, which then made solving the linear system actually the fastest part, much faster than building the predictor matrices in the first place - to some extent because I was using, and too lazy to avoid using, MATLAB For loops. I eventually switched over to building the predictors in Python. So this is my story, hope you liked it.
|
# ? Mar 28, 2015 17:29 |
|
I don't think mldivide is as relevant to GLM as it is to simple regression. GLM is usually solved using an iterative algorithm. In any case, statsmodels does use UMFPACK when it can. At least it includes it, so I assume it uses it. Not sure if it is actually helpful in the case of GLM though. GLM can be slow. If you really need GLM and not simple regression you may be out of luck. You could benchmark other GLM solvers in R and Matlab but I doubt it would make much difference.
|
# ? Mar 28, 2015 19:07 |
|
I'm stumped. Trying to learn the very basics of databases, using sqlite and python (playing with perl too). I have a simple 1219-row database that has a table called patients with 4 columns: code:
Python code:
Python code:
Hughmoris fucked around with this message at 04:50 on Mar 29, 2015 |
# ? Mar 29, 2015 04:46 |
|
Instead of executemany, try using execute. Also, you should use the row own id instead of a counter to put in the WHERE clause. Also, if you really want to use a counter (you shouldn't in this case), you should take a look at the enumerate built in. edit: don't do that.
|
# ? Mar 29, 2015 04:53 |
|
HardDisk posted:Instead of executemany, try using execute. What do you mean by : Also, you should use the row own id instead of a counter to put in the WHERE clause. Yeah, I figure there has be a better way to iterate over rows but I'm not sure how. I used execute instead of executemany in my original code, no luck.
|
# ? Mar 29, 2015 04:57 |
|
Your each_patient variable should contain information about the row that you are iterating on. If I had to guess, it returns a dictionary mapping each column to a key; You would then access the id value by doing each_patient['Id'] and then you would pass that to execute. For examplePython code:
I'm still not convinced that the executemany wasn't the culprit. See: https://docs.python.org/2/library/sqlite3.html#sqlite3.Cursor.executemany What happens when you print each_patient? Space Kablooey fucked around with this message at 05:09 on Mar 29, 2015 |
# ? Mar 29, 2015 05:06 |
|
Hughmoris posted:What do you mean by : Also, you should use the row own id instead of a counter to put in the WHERE clause. Your insurance carrier table should be something like code:
But really, grab a book about database design before you end up with an abomination. If you don't have the time for that, grab an ORM like http://sqlalchemy.readthedocs.org/en/latest/orm/tutorial.html -- that should make it easier to model what you want if you already know python.
|
# ? Mar 29, 2015 05:12 |
|
Hughmoris posted:I'm stumped. Trying to learn the very basics of databases, using sqlite and python (playing with perl too). Dude, please tell me that those are not real names and social security numbers
|
# ? Mar 29, 2015 08:45 |
|
They're not. From the General Programming Questions thread:Hughmoris posted:Well, I decided to try my hand at this to learn a little bit about databases. Using Python (did it in perl too cause I'm bored) I created a database, made a table and populated it with first name, last name, SSN and insurance carrier. I found text files with common first and last names, randomized then combined those. Same with insurance carriers, and I created a SSN generator. Its sitting at 1219 rows right now.
|
# ? Mar 29, 2015 14:21 |
|
QuarkJets posted:Dude, please tell me that those are not real names and social security numbers I'm not a smart man, but I'm smarter than that.
|
# ? Mar 29, 2015 14:31 |
|
I need to take a video and extract the video frame when a certain audio pattern is detected, a loud predictable beep. The Echo Nest (from this a while back) looks a lot more suited to dealing with music videos or other popular content and I'd rather not set up an API key for this goofy one-off experiment. Can anyone point me to a library or something similar that would make this easier?
|
# ? Mar 31, 2015 00:29 |
|
I'm going to be using Python to talk to a PostgreSQL database and deliver up records as an API for a website. I've also decided that I'll work directly with the database, forgoing ORMs and just dealing with raw SQL calls. I still need user authentication and authorization, though, and I'm not sure where I should turn for that. I'm leaning towards webpy for the actual API infrastructure but it doesn't appear to have anything to deal with authentication so I'd still need to find something to handle tokens and whatnot. Is there a simple framework for dealing with stuff like this? Edit: Maybe Flask would be better for this situation? IAmKale fucked around with this message at 02:24 on Mar 31, 2015 |
# ? Mar 31, 2015 02:10 |
Karthe posted:I'm going to be using Python to talk to a PostgreSQL database and deliver up records as an API for a website. I've also decided that I'll work directly with the database, forgoing ORMs and just dealing with raw SQL calls. I still need user authentication and authorization, though, and I'm not sure where I should turn for that. I'm leaning towards webpy for the actual API infrastructure but it doesn't appear to have anything to deal with authentication so I'd still need to find something to handle tokens and whatnot. Is there a simple framework for dealing with stuff like this? Why not Django + Django Rest Framework? You get the auth, ORM, API, schema migrations, admin interface, etc all for free. It's easy to customize the DRF API endpoints to your liking, and you can add additional ones really easily. You can ignore all the views/forms crap built into Django.
|
|
# ? Mar 31, 2015 02:55 |
|
Flask or bottle would be fine, but yeah Django with this would be pretty easy as fletcher said.
|
# ? Mar 31, 2015 04:00 |
|
generator.py:code:
code:
pre:C:\Users\Documents>python generator.py one Tue Mar 31 10:39:51 2015 two Tue Mar 31 10:39:52 2015 thr Tue Mar 31 10:39:53 2015 C:\Users\Documents>python generator.py | python filter.py ONE TUE MAR 31 10:40:02 2015 Tue Mar 31 10:40:05 2015 TWO TUE MAR 31 10:40:03 2015 Tue Mar 31 10:40:05 2015 THR TUE MAR 31 10:40:04 2015 Tue Mar 31 10:40:05 2015 What I want: pre:C:\Users\Documents>python generator.py one Tue Mar 31 10:39:51 2015 two Tue Mar 31 10:39:52 2015 thr Tue Mar 31 10:39:53 2015 C:\Users\Documents>python generator.py | python filter.py ONE TUE MAR 31 10:40:02 2015 Tue Mar 31 10:40:02 2015 TWO TUE MAR 31 10:40:03 2015 Tue Mar 31 10:40:03 2015 THR TUE MAR 31 10:40:04 2015 Tue Mar 31 10:40:04 2015
|
# ? Mar 31, 2015 15:42 |
|
epalm posted:generator.py: Haven't tried it, don't know much about this sort of stuff, but is there some way of "flushing" the output of generator.py using the sys.stdout file-like object? It does have a flush() method.
|
# ? Mar 31, 2015 15:49 |
|
Hammerite posted:Haven't tried it, don't know much about this sort of stuff, but is there some way of "flushing" the output of generator.py using the sys.stdout file-like object? It does have a flush() method. I don't actually have control over generator.py, it's another program spitting out log files. For the sake of argument, I added sys.stdout.flush() after every print call, and there was no change in behavior.
|
# ? Mar 31, 2015 15:57 |
|
Hed posted:Flask or bottle would be fine, but yeah Django with this would be pretty easy as fletcher said. That said, what exactly does a web-facing Python host look like? As in, I'm familiar with the LAMP stack but I'm still not sure how Python and Apache work together to enable me to execute Python like I would PHP. I know that Python's WSGI functionality has something to do with it, and that there's such a thing as a webserver with WSGI support. I just don't know how to build a server that'll play API host with a Python backend. Is Apache still needed if frameworks like Flask can handle HTTP connections? Do I just to expose a port and run Flask on that port?
|
# ? Mar 31, 2015 17:16 |
|
Karthe posted:I have to be honest, I was out of the web dev game for so long that I'm scrambling to get back to up to speed for work. At the moment I've got enough on my plate learning AngularJS as well as Single-Page-App architecture and all the craziness that entails (the last time I did any web app development it was still cool to render stuff on the server). I'm just looking for something I can bang on and get an API up and running, so I think I'll roll with Flask and Flask-JWT for right now and look into Django-Rest-Framework later on down the road. I think what they're saying is that the Django way is the easiest way. You're choosing the harder method.
|
# ? Mar 31, 2015 17:22 |
|
epalm posted:I don't actually have control over generator.py, it's another program spitting out log files. readlines() reads until EOF, and splits by lines. What you are actually iterating over is the entire input as a list. Something like this could work: code:
|
# ? Mar 31, 2015 17:33 |
|
Karthe posted:I have to be honest, I was out of the web dev game for so long that I'm scrambling to get back to up to speed for work. At the moment I've got enough on my plate learning AngularJS as well as Single-Page-App architecture and all the craziness that entails (the last time I did any web app development it was still cool to render stuff on the server). I'm just looking for something I can bang on and get an API up and running, so I think I'll roll with Flask and Flask-JWT for right now and look into Django-Rest-Framework later on down the road. You do something like this: http://flask.pocoo.org/docs/0.10/deploying/mod_wsgi/ if you're going to go with Flask + Apache. Something like Django is way easier to use quickly though.
|
# ? Mar 31, 2015 17:34 |
|
Thermopyle posted:I think what they're saying is that the Django way is the easiest way. You're choosing the harder method. EAT THE EGGS RICOLA posted:Something like Django is way easier to use quickly though.
|
# ? Mar 31, 2015 17:47 |
|
Pie Colony posted:readlines() reads until EOF, and splits by lines. What you are actually iterating over is the entire input as a list. Or Python code:
|
# ? Mar 31, 2015 18:45 |
|
fanstatic is driving me insane okay well thanks for listening
|
# ? Mar 31, 2015 18:53 |
|
Thanks for convincing me to use DRF, I've been pounding away at it today and it wasn't as painful as I thought it would be. I even found a great JWT plugin so I can use them for session management. I still have a lot to learn about DRF's models and views (and I suppose ORMs in general) but at least now I have a functioning API set up to play around with.
|
# ? Apr 1, 2015 01:14 |
|
EAT THE EGGS RICOLA posted:You do something like this: http://flask.pocoo.org/docs/0.10/deploying/mod_wsgi/ if you're going to go with Flask + Apache. It's been a few years since I did anything with Django but: it is that easy to just put Django in front of an arbitrary database and use it as the ORM for the site? I seem to recall Django being a bit pernicky about how db's were laid out. Of course, it depends on how complex the object are that you are pulling out and how polluted my memory is by all those frameworks that are really strict about database structure ("must be named foo_id and must be an integer ...")
|
# ? Apr 1, 2015 12:52 |
|
JawnV6 posted:I need to take a video and extract the video frame when a certain audio pattern is detected, a loud predictable beep. The Echo Nest (from this a while back) looks a lot more suited to dealing with music videos or other popular content and I'd rather not set up an API key for this goofy one-off experiment. Can anyone point me to a library or something similar that would make this easier? I don't know of any libraries, but it's it's just a beep maybe the Goertzel algorithm would work? Just take the audio stream, send it thru, and look for where you have energy in that bin? (http://en.wikipedia.org/wiki/Goertzel_algorithm http://www.embedded.com/design/real-world-applications/4401754/Single-tone-detection-with-the-Goertzel-algorithm)
|
# ? Apr 1, 2015 12:54 |
|
outlier posted:It's been a few years since I did anything with Django but: it is that easy to just put Django in front of an arbitrary database and use it as the ORM for the site? I seem to recall Django being a bit pernicky about how db's were laid out. Of course, it depends on how complex the object are that you are pulling out and how polluted my memory is by all those frameworks that are really strict about database structure ("must be named foo_id and must be an integer ...") Django is not just a ORM, though, it a really complete web framework that happens to be somewhat opinionated about the structure of your code. Not that it is a completely bad thing, because if you are just starting with web dev, it is easy to pick it up and let it run. Your business logic shouldn't be constrained by your web framework of all things. If you don't like Django's opinions, then there are alternatives like Flask, where it has very little say on how you should do things and it is very extensible, but it is easier to make a mess of your code.
|
# ? Apr 1, 2015 14:04 |
|
SurgicalOntologist posted:Or Did you even try your code? It doesn't do the same thing.
|
# ? Apr 1, 2015 15:35 |
|
My bad, I meant to be demonstrating iteration over sys.stdin. The change from print to sys.stdout was incidental and I forgot the newline added by print. I also forgot to combine the strings.
|
# ? Apr 1, 2015 16:31 |
|
I forgot to mention, if you want to do simple in memory caching without the hassle of setting up memcache or redis, I found a package called 'expiringdict' which is essentially that - a dict that expires and handles all the locking stuff for you. Also, Yahoo just released redislite, which is redis without a server. This will be really nice for those that need to use Redis in production, and don't want to set it up on their dev box. https://github.com/yahoo/redislite
|
# ? Apr 1, 2015 18:06 |
|
I'm writing a bunch of tests at work, and I'll need to compile a small table documenting my tests. I figured it would be easiest just to document my tests in the docstring of each test, but what do you guys think the best method of pulling these out would be? I haven't used it before, but sphinx seems like it would be a good candidate. Looking at the documentation, it seems like this should be easy enough to pull off but I'm kinda lost with how to start. How do I go about creating a simple page that will have a table populated by particular items from my docstring?
|
# ? Apr 1, 2015 20:15 |
|
Karthe posted:JWT plugin Hmm, I've read that phrase today... where was - oh https://auth0.com/blog/2015/03/31/critical-vulnerabilities-in-json-web-token-libraries/ Careful!
|
# ? Apr 1, 2015 21:04 |
|
SurgicalOntologist posted:My bad, I meant to be demonstrating iteration over sys.stdin. The change from print to sys.stdout was incidental and I forgot the newline added by print. I also forgot to combine the strings. What I meant was that using sys.stdin's iterator works differently than repeatedly calling readline().
|
# ? Apr 1, 2015 21:52 |
|
Oh. I didn't realize. They behave the same for me (after fixing the sys.stdout.write thing). Maybe it's a Python 2 vs. Python 3 thing? I only tested on 3.
|
# ? Apr 1, 2015 22:13 |
|
Repeatedly calling readline in an infinite loop would work even after the stream is closed and reopened, if that was to happen somehow, while the iterator ends when the stream closes or if there is open stream. I'm not aware of any other significant differences between the two. I suppose either behavior could be desired.
|
# ? Apr 2, 2015 00:57 |
|
You mean without the if not line: break piece, right? Or does it not send an empty line if the stream is closed?
|
# ? Apr 2, 2015 02:10 |
|
Karthe posted:I have to be honest, I was out of the web dev game for so long that I'm scrambling to get back to up to speed for work. At the moment I've got enough on my plate learning AngularJS as well as Single-Page-App architecture and all the craziness that entails (the last time I did any web app development it was still cool to render stuff on the server). I'm just looking for something I can bang on and get an API up and running, so I think I'll roll with Flask and Flask-JWT for right now and look into Django-Rest-Framework later on down the road. Its *still* cool to render on the server. The whole serve-a-blob-of-js style of web development is hopelessly misguided in my opinion. Sure google parses javascript these days, but its not that bright about it, and its still going to favor HTML5 over "poo poo all over the browser with javascript". And yeah AngularJS is insufferable. Check out Ractive, hook it up with JQuery, and it'll do 90% of what you'll want out of Angular, with a half-day learning curve instead of a few weeks of tearing your hair out fighting with Angulars bizzare and opaque ways. If I never have to develop an "enterprise" Angular app again, I'll be a very happy man. edit:
|
# ? Apr 2, 2015 04:32 |
|
|
# ? May 9, 2024 07:52 |
|
duck monster posted:Its *still* cool to render on the server. The whole serve-a-blob-of-js style of web development is hopelessly misguided in my opinion. Sure google parses javascript these days, but its not that bright about it, and its still going to favor HTML5 over "poo poo all over the browser with javascript". I think you meant React instead of Ractive (probably not, but I'm still saying it). But yeah, you're right about Angular being opinionated and not fun to work in. Anyone doing That's not why we're going to a thinner rendering layer however. We have spent thousands of dev hours migrating from WebForms to .NET MVC, building our own MVC templating engine before Razor was a thing, building our own node.js framework, moving to the cloud etc. always being chained to years of technological cruft. I'm sure there will be new things that bite us in the rear end, but we're treating the server as an API that returns JSON, and the web framework is going to essentially just handle routing, caching, and a few other middleware type things so we have flexibility going forward. luchadornado fucked around with this message at 17:55 on Apr 2, 2015 |
# ? Apr 2, 2015 05:30 |