|
sharktamer posted:Is this easy enough to use with flask as is? If so, would you be willing to add some use cases to the user guide? Absolutely, you can currently serve up both static pages pages or pages with server-backed plots. We're in the process of adding an "HTML Fragment" session that will make this even easier. Anyway one of my colleagues just offered to work up a simple standalone Flask example and add it to the user guide so hopefully I can point you to something concrete in the next few days.
|
# ? Mar 11, 2014 16:04 |
|
|
# ? Jun 13, 2024 06:08 |
|
I'm writing some unit tests for a script that parses an html page. For testing, I'm using a reference HTML file that's a full WGET of the target page. What's best practice for the assert against a function that returns all tags according to a selector, like lxml's "cssselect()" or bs4's "find_all()"? Let's say that the reference page is supposed to return 14 <a> items as a list. Is it enough to just verify the "len()" of the results, just check a few of the actual values (maybe first and last), or verify that the full result list matches my list of expected results? The answer might be "it depends," and that's ok. Mostly I'm wondering if the either or both of first two methods would be considered insufficient or bad practice.
|
# ? Mar 11, 2014 19:37 |
|
Why do you have it in a special function? Isn't returning all the tags matching a selector what find_all() does? Why would you stuff that behavior in a new function then try to test it?
|
# ? Mar 11, 2014 21:30 |
|
Dren posted:Why do you have it in a special function? Isn't returning all the tags matching a selector what find_all() does? Why would you stuff that behavior in a new function then try to test it? Maybe I oversimplified the example or maybe I'm over-testing. In actual practice, it would only be a function becasue it's going to be called multiple times and has conditionals. Maybe a better example would be a function that returns the urls of leeched images found on a page. (I have a couple of clients that I've been unable to break of the habit when they make blog posts.) So a "leeched_images(url)" function might return 10 of 14 found <img> on site A, and 1 of 6 <img> on site B. Super-hacky pseudo-code below. If the right thing to do is test the "img_is_on_host(imgsrc)" function and not test "leeched_images()" at all, that's fine. Just trying to understand where to draw the line on testing. Python code:
onionradish fucked around with this message at 23:00 on Mar 11, 2014 |
# ? Mar 11, 2014 22:57 |
Plorkyeran posted:Build wheels for all of your packages? Got wheels built for everything, yielded a huge improvement in how long it takes to deploy the app! Thanks!
|
|
# ? Mar 12, 2014 00:34 |
|
salisbury shake posted:
Gonna resurrect this from the bottom of the last page
|
# ? Mar 12, 2014 03:07 |
|
Onionradish, I would consider what you're doing to be more integration testing than unit testing because you are running sample input through the entire routine rather than chunking up the routine into small pieces and verifying a smaller subset of behaviors.
|
# ? Mar 12, 2014 04:37 |
|
The thing about unit testing is that you want to structure your code so that you are only testing inputs and outputs of functions. The idea being that if you can trust those inputs and outputs, everything will work. So for your code the appropriate testing would be to test leeched_images() as a whole. You'd want to see that for some input url you get the proper list of leeched images as a result. You could check the len of the list and verify that the actual results are as expected. It sounds like you want finer grained testing, to do that, break your code up into more functions like so: Python code:
|
# ? Mar 12, 2014 16:59 |
|
Thanks for taking the time to write that out -- it was really helpful.
|
# ? Mar 12, 2014 17:51 |
|
Today I spent some time testing out some scripts that fit a lot of linear models. I'm using statsmodels for the formula building and fitting and pandas for the data. I started off using my generic Anaconda environment which has all of the Accelerate packages installed, like MKL numpy. When I fit my model (~250 data points, 4 predictors) it uses all 16 threads on my processor and takes about 250ms. This seemed kind of slow so I compared it to the standard numpy environment to see if my model fitting is just really really slow that way. It turns out standard numpy performs the same fit in 60% of the time using only one thread. With Anaconda Accelerate enabled with 16 threads. Maxes out all 16 threads for the duration of the test. Python code:
Python code:
|
# ? Mar 12, 2014 18:36 |
|
salisbury shake posted:Gonna resurrect this from the bottom of the last page Well, quite simply it doesn't work. Your setup.py script has a reference to some 'AwfulGUI_old' package that doesn't exist. Then when I remove that to get the setup.py working, I get the following when trying to create an AwfulPy instance like in your README.md. It's entirely possible it's me doing something stupid though, but I think this might be a rare case where it's not.
|
# ? Mar 12, 2014 18:47 |
|
Yea, I should delete the setup.py, it was for initializing the PyPi repo. Has nothing to do with setting anything up. Accidentally leaked py3k super() usage into py2k compatible code . You should be able to run it with python3 or ipython3 and hopefully that'll be the only issue you run into. Found a couple of bugs in the search that uses the old search.php and lastread, though. Thanks for the feedback and taking the time to post get the whole traceback
|
# ? Mar 13, 2014 01:20 |
|
OnceIWasAnOstrich posted:Today I spent some time testing out some scripts that fit a lot of linear models. I'm using statsmodels for the formula building and fitting and pandas for the data. I started off using my generic Anaconda environment which has all of the Accelerate packages installed, like MKL numpy. When I fit my model (~250 data points, 4 predictors) it uses all 16 threads on my processor and takes about 250ms. This seemed kind of slow so I compared it to the standard numpy environment to see if my model fitting is just really really slow that way. It turns out standard numpy performs the same fit in 60% of the time using only one thread. Your models are very small so I'm not surprised you're losing time to overhead from multiple cores. Try using the anaconda + accelerate with MKL_NUM_THREADS=1 set in your environment; this will use MKL but will tell it to only use one thread. If your models are independent then consider parallelizing over models instead of inside the optimization. I like to use joblib for this since it has a very nice and painless interface for executing task-parallel jobs using multiprocessing.
|
# ? Mar 13, 2014 01:38 |
|
Nippashish posted:Your models are very small so I'm not surprised you're losing time to overhead from multiple cores. Try using the anaconda + accelerate with MKL_NUM_THREADS=1 set in your environment; this will use MKL but will tell it to only use one thread. If your models are independent then consider parallelizing over models instead of inside the optimization. I like to use joblib for this since it has a very nice and painless interface for executing task-parallel jobs using multiprocessing. Thanks, it turns out there is zero speedup with MKL over normal numpy in this particular application, but at least now I don't need to create a virtualenv or whatever the conda equivalent is to prevent a 2-fold slowdown. I've been using the map function of multiprocessing Pools to parallelize the normal version, which seems a lot simpler than bringing in a whole other library to do the same thing with extra syntax and imports. I guess it might help with the ability to automatically memmap arrays but I am already using HDF5 which works great with read-only multiprocessing.
|
# ? Mar 13, 2014 01:54 |
|
OnceIWasAnOstrich posted:I've been using the map function of multiprocessing Pools to parallelize the normal version, which seems a lot simpler than bringing in a whole other library to do the same thing with extra syntax and imports. The embarrassingly parallel for loops thing I linked to essentially does this but with less boilerplate. If you're already set up using multiprocessing directly then there is not much point to switch to something new though. I just mention it because I like their interface a lot more than using multiprocessing directly.
|
# ? Mar 13, 2014 10:44 |
|
Thanks in advance for reading - this is kind of long and confusing... I have a reasonably complex logging system I need to implement for an application I'm writing and I'm having a hard time finding REAL use cases that I can steal best practices from. Some help would be really appreciated... First, it's probably useful to understand the application - apologies for verbosity, but it's required to see why the logging requirements are a little complicated. At the most basic level it's basically a secondary init in that its main purpose is to start and stop other applications. The difference is that these can be installed in user space and run within their own security context and don't require root to install or run. Each user who has scripts to run will create a special dot-directory in their $HOME, and within that directory is a manifest of all the "packages" they want to execute ('packages.ini') along with one directory per entry containing everything that is required to run (or a gzipped tarball of such a directory, called a .pip, which is how I'll refer to them from now on). Each pip has a settings.ini file which defines all sorts of things like the entry point and arguments for the script, etc. Also in this file is the log entry string format, filename (or handler type, potentially) and logging.LEVEL that the person who wrote that script wants to use. The main application lives in myApp.py, and all the special classes (CommandBase, Package, Manifest) I've created live in lib/myApp_lib.py. CommandBase is a class from which all other application classes inherit, and it contains the master level ConfigParser crap to pass around to all the various other classes. This is important later. I need to log the following:
My confusion is stemming from the fact that each pip needs to log to two places at two (possibly) different verbosenessesseses, and the master application is also logging its own stuff. What I'm trying to figure out is how many loggers I need, where they should be instantiated, and how I should be passing log messages around from my derived classes. Should I create the Logger object inside CommandBase? In this case every instance of any class that inherits it should have its own Logger (which means one Logger per pip, as each pip is its own object - good!). Or should I create the Logger objects inside every class that I want logging for? If so, how should they be named? I can't simply use getLogger(__name__) because then the Manifest object and Package object will have different Loggers associated and that's not what we want. My best guess is that I should create a master Logger either in main() or in __init__() for the myApp class, then the child Logger will go into the Package class so there'll be one logfile per package. But then I'm stuck at that point on how to get the package log to write to two places at two LEVELs. I'm also not sure how to get info from the other classes to be written to these logs (especially information from libraries I didn't write) - should I just be catching Exceptions and Warnings and passing them into the logs with try/except blocks? I know this is really confusing if you don't have access to my code, which is why I'm hoping someone can offer some guidance and point me to some public code somewhere that makes use of the standard logging class and then I can figure it out from there. If it's helpful I may be able to write out some skeleton code to describe the structure a bit clearer - let me know if that's useful. Thank you so, so, SO much
|
# ? Mar 14, 2014 20:17 |
|
scholzie posted:Thanks in advance for reading - this is kind of long and confusing... If your code is heavily OO then you might look into the mixin concept. You could create a mix-in class for each discrete log file you want to have, and then inherit from the proper logging class in each class you want to do logging in. Here's a rough sketch that I think will work but I haven't tested it: Python code:
Hopefully this makes some sense.
|
# ? Mar 14, 2014 21:21 |
|
I'm curious, why reinvent the wheel? There are already a lot of tools out there that exist for scheduling and running other tools. For example, just toss up a CI server and let people create their own jobs with their own logging configurations within it.
|
# ? Mar 14, 2014 23:32 |
|
Not sure this is the best place to ask, but there didn't seem to be a Qt framework thread. I'm trying to build a basic file browser in PyQt, using a QTreeView for the directory list, and a separate QListView to show the appropriate image files in that folder. Problem is, when I use setRootPath() to change the shown directory in the QListView, to the one selected in the QTreeView, it somehow messes with the filters, and the ListView will show the system drives instead of the directory contents. I'm using QDir.Files | QDir.NoDotAndDotDot to filter the QListView, and originally, there was an issue with directories showing up despite these filters, too, but I've solved that by resetting the filters after each setRootPath(), but that hasn't helped the drives to not randomly appear. And now that I've typed that out, I'm not sure why I didn't use QFileDialog and save myself a whole lot of work, except that I don't think it'll quite exactly follow the behaviour I'd like to achieve.
|
# ? Mar 15, 2014 11:22 |
|
Surprise T Rex posted:Not sure this is the best place to ask, but there didn't seem to be a Qt framework thread. Do you want the same filters applied to both the QTreeView and QListView? In the past, I have used the same QFileSystemModel for both Tree and List Views. I maintain location compatibility between the two views by adjusting for each tree/list view setRootIndex(fileSytemModel->index(path)).
|
# ? Mar 15, 2014 15:39 |
|
I need the TreeView to only show directories and the ListView to only show files, so using the same FileSystemModel seems like it wouldn't work, unless I'm missing a detail. I did try implementing a QFileDialog instead, but I'd have to mess around with it a bit anyway, since the views aren't quite how I'd like them.
|
# ? Mar 16, 2014 18:36 |
|
3.4.0 is out
|
# ? Mar 17, 2014 14:17 |
|
Does that mean python comes with pip included now?
|
# ? Mar 17, 2014 14:47 |
|
Well, it looks like Python will finally be getting a new infix operator suitable for matrix multiply! https://groups.google.com/forum/#!topic/python-ideas/aHVlL6BADLY%5B51-75-false%5D
|
# ? Mar 17, 2014 15:15 |
|
sharktamer posted:Does that mean python comes with pip included now? Yes, and as an added bonus, the global pip3.4 respects any active pyvenv-3.4 virtualenv. In 3.3 on Debian and Ubuntu, I had to install pip itself in the virtualenv since the system pip would try to write to /usr/local/lib/python3.3. (The old virtualenv package automatically installs pip and setuptools into virtualenvs as they're created.)
|
# ? Mar 17, 2014 15:21 |
|
If there are any Ontarians heading to Pycon next month, check out http://pythononrails.ca/ for a really cool way to get there.
|
# ? Mar 17, 2014 18:13 |
|
I'm finally trying to learn a language properly, and python it is, since I found "Learn Python the hard way", which suits my learning style well, online. Wanting some training material i found 46 Simple Python Exercises : http://www.ling.gu.se/~lager/python_exercises.html that I'm enjoying very much so far. Looking forward, I find this, under Simple exercises including I/O : quote:The International Civil Aviation Organization (ICAO) alphabet assigns code words to the letters of the English alphabet acrophonically (Alfa for A, Bravo for B, etc.) so that critical combinations of letters (and numbers) can be pronounced and understood by those who transmit and receive voice messages by radio or telephone regardless of their native language, especially when the safety of navigation or persons is essential. Here is a Python dictionary covering one version of the ICAO alphabet: Perfect. (Under UNIX/Linux and Windows, something similar might exist.) Fun, and challenging, in the way that Python has it's own runtime that isn't perfectly aligned with .Net or COM or CLR or whatever they are called deep down in the Windows engine room that powers my internet vehicle. Now, part of my motivation for learning Python is that I've read it's popular as a "glue" language. I had my first sniff of glue when I was thirteen, and I never did it again, beacuse the effects weren't that hilarious, but on the other hand gluing can't do much damage. In fact it's probably gluing I want to learn more than anything else. Or as they call it : seamless integration. I've done some decent problem solving and model building using Excel, Matlab, Mathematica, VBA, C#, VB and whatnot (but always task oriented, fixing something urgent now) so I'm pretty sure I can solve the "speak" part with a given string in a .Net environment using one of the text to speach libraries. However - the initialising - the handshaking - the sending of the string, or the charachers ( in a stream?) (over a socket or a port?) - the confirmation etc. Making two runtimes speak and solve a problem together is what I want to learn and why this excersice is perfect. So, where do I start ? (I consider IronPython a workaround, I'm learning the hard way)
|
# ? Mar 17, 2014 20:45 |
|
Write a function in whatever language you want to do the "speak" part and then use os.system to call it. With this approach you would use python to generate the arguments to pass to your other function.
|
# ? Mar 17, 2014 22:22 |
|
For Windows, you should be able to call the Microsoft Speech API (SAPI) from Python. I saw this page: http://stackoverflow.com/questions/15879802/python-win32com-client-geteventssapi-spsharedrecocontext-returns-none I haven't ever tried calling any COM stuff with Python, it just seemed like a possibility so I googled for it.
|
# ? Mar 17, 2014 23:02 |
|
http://code.activestate.com/recipes/578839-python-text-to-speech-with-pyttsx/ ? or https://gist.github.com/alexsleat/1362973
|
# ? Mar 18, 2014 02:58 |
|
Maybe the Python thread is a better place for this I'm thinking about writing a crappy GUI for some of my Ansible playbooks, so I can have other people execute them with a button or something. Any ideas on the best way to accomplish this? I was thinking of using django and keeping some light data like last execution etc and just using os.system. Are there any libraries or frameworks out there that work well with executing stuff like this?
|
# ? Mar 18, 2014 02:59 |
|
pram posted:Maybe the Python thread is a better place for this
|
# ? Mar 18, 2014 05:10 |
|
I may be misunderstanding what you need, but what about Ansible Tower http://www.ansible.com/tower
|
# ? Mar 18, 2014 05:11 |
|
Misogynist posted:Any reason you'd roll your own instead of using something like RunDeck? Hmm interesting, I considered Jenkins but this looks better. Thanks! more like dICK posted:I may be misunderstanding what you need, but what about Ansible Tower http://www.ansible.com/tower I've seen this but the pricing is ridiculous.
|
# ? Mar 18, 2014 06:04 |
|
I think I know the answer to this but I thought I'd ask you guys anyway. I've been writing some crap that makes use of the jsonselect module. I've decided that I'd like to build a curses interface that requires the npyscreen module. The problem is npyscreen, while it 'supports' python 2.7, doesn't work properly unless I'm using python 3.3. Unfortunately, jsonselect doesn't support python 3 at all. I think the answer is, 'be a better coder and stop using all these lovely modules'. Unfortunately I'm time and intellect limited so I'm thinking of just reverting back to python 2.7 (the crux of my code is to take two big json files, find a specific leaf in each json, calculate a diff) and figuring out something else for my interface. What would you guys do?
|
# ? Mar 18, 2014 06:55 |
|
It appears there are a ton of JSONPath projects. Maybe try jsonpath-rw instead? https://pypi.python.org/pypi/jsonpath-rw. It says it works in 2.7 and 3.3.
|
# ? Mar 18, 2014 15:32 |
|
pram posted:Maybe the Python thread is a better place for this We use Jenkins for this, which is kind of nice since that way they get run off a central server that stores the logs of each run, multiple people can watch a run in progress, the build doesn't abort if you close the window, etc. You can also set up about 9,000 notification options.
|
# ? Mar 18, 2014 16:51 |
|
I'm setting up CI for a project, first time using Python in that context and I'm wondering what approach is best. From my brief investigation so far it seems like tox can be fed my requirements.txt so I can spin up a virtualenv and run tests in the QA phase, and after that, deploying should I just use tox on the deployment target? Or should I be using pex or wheels or something like that. In most cases I'll definitely be deploying to another Linux server, but not always sure whether it'll be rpm or Debian. Any advice from people who've done it before?
|
# ? Mar 20, 2014 05:29 |
|
Maluco Marinero posted:I'm setting up CI for a project, first time using Python in that context and I'm wondering what approach is best. If you're deploying to multiple environments, not just using different python versions, I don't think tox is really enough for you. It sounds like you just need separate testing/integration environments for each target. Maybe a build slave for each?
|
# ? Mar 20, 2014 17:25 |
|
|
# ? Jun 13, 2024 06:08 |
|
I have a question about the interpreter for y'all. I'm running a long running task an an Arduino. The task itself is not relevant, but it periodically sends status updates over serial, which I am picking up with a python script on my raspberry pi, and echoing across jabber so they hit my phone. The problem is, the python script is killing itself after about 20-30 hours. It must be a problem with the script because I've run the program for 48 hours on my desktop using the arduino IDE serial monitor and the arduino is fine. Does the python interpreter have some kind of "you've been running in a constant loop too long I'm going to you" built into it?
|
# ? Mar 21, 2014 08:00 |