|
I've got a little puzzle... we have a bunch of Python packages with a similar test setup: pytest with pytest-cov, pytest-flake8, pytest-docstyle, and pytest-mypy. We just copied this configuration into a new library, one that has zero actual tests at this point (for the purpose of getting at least the linting for now), and we got 49% coverage. What gives? Does one of these linters actually execute the code in a way that registers the coverage measurement? And if so why only 49%? We're all quite confused.
|
# ? Oct 10, 2019 16:34 |
|
|
# ? May 27, 2024 02:43 |
|
Very simple question, I have a bunch of files like 1.json, 2.json, and so on that I need to play with in that order. I try to use this: code:
|
# ? Oct 12, 2019 20:57 |
|
You could zero pad all of the file names that start with a number prior to sorting
|
# ? Oct 12, 2019 21:07 |
|
If you can't rename them then a for i in range(len(file_list)) and constructing the filename from i+1 would be easy, assuming no numbers are skipped in the filenames.
|
# ? Oct 12, 2019 21:19 |
code:
|
|
# ? Oct 12, 2019 21:37 |
|
Thanks, I ended up padding with zfill.
|
# ? Oct 12, 2019 21:49 |
|
https://www.vice.com/en_us/article/zmjwda/a-code-glitch-may-have-caused-errors-in-more-than-100-published-studies?utm_source=reddit.com The original paper can be found here: https://pubs.acs.org/doi/10.1021/acs.orglett.9b03216 If you dig enough you can find out that they assumed glob.glob output was sorted.
|
# ? Oct 13, 2019 03:02 |
|
Natural sort: https://stackoverflow.com/questions/4836710/does-python-have-a-built-in-function-for-string-natural-sort
|
# ? Oct 13, 2019 03:22 |
|
I am reading my way through the testing goat book right now. At some point he recommends automating deployment using: code:
When I googled it it looks like there is no option or even variant to automatically create a venv with a python version that isn't already installed through the package manager. To me that implies that everybody in the python world expects that their software will be utterly abandoned by all users by the time 3.8 becomes the default for new linux installs. Is that really a reasonable assumption in the python world?
|
# ? Oct 13, 2019 16:16 |
|
I think the assumption is you must have the desired Py version installed. On Linux or Mac, this is straightfwd to do by building from source; using the included instructions, it will install to `usr/bin` under the alias `python3.7` etc. You can then either run `python3.8 -m venv virtualenv`, or `./usr/bin/python3.8 -m venv virtualenv`. (You may be able to find unofficial package-manager versions, but these are distro-specific, and may-or-may not be a pain to install) In Windows, the best way may be to use the installers; I don't think you'll get the aliases, but can use `./C/Program files/python3.8/Scripts/python.exe` etc.quote:When I googled it it looks like there is no option or even variant to automatically create a venv with a python version that isn't already installed through the package manager. VictualSquid posted:To me that implies that everybody in the python world expects that their software will be utterly abandoned by all users by the time 3.8 becomes the default for new linux installs. Dominoes fucked around with this message at 16:38 on Oct 13, 2019 |
# ? Oct 13, 2019 16:25 |
|
Thermopyle posted:https://www.vice.com/en_us/article/zmjwda/a-code-glitch-may-have-caused-errors-in-more-than-100-published-studies?utm_source=reddit.com
|
# ? Oct 13, 2019 16:36 |
|
Yeah. It was actually caught because of different results in the same lab on computers running different OS's.
|
# ? Oct 13, 2019 16:39 |
|
Dominoes posted:I think the assumption is you must have the desired Py version installed. Yea, auto installing the latest python version for an environment of an otherwise known config seems like begging for problems.
|
# ? Oct 13, 2019 19:01 |
|
Dominoes posted:I haven't dug into the original chem papers, but it's surprising this wasn't picked up earlier due to surprising or inconsistent results... it makes me wonder how much publication bias played in. It's a small miracle that it was discovered at all The fundamental issue was that whoever wrote the script assumed that glob provides sorted results, so the returned file ordering was OS-dependent. Someone who's not actively looking for bugs in the script likely wouldn't notice this assumption, and in fact, hundreds of researchers using the script didn't notice the issue for over 5 years; how often does a chemist perform code review for widely-used software that they didn't write? I doubt it ever happens. And then this error produces output that still looks entirely reasonable; in the example provided, one OS produced a value of 173.2 while another produced 172.7. A lot of the studies suffering from the problem might not even be sensitive to a calculation bias this small. And many of those studies actually published correct results, because they were running on an OS where glob coincidentally provided results in the order that the script was expecting We like to imagine that scientists are rigorously reverifying each others' work all the time, but the fact of the matter is that there's little fame or profit to be found in doing that kind of essential work, so this reverification process doesn't happen nearly as often as it needs to. And in this reverification study, if the student had been running on an OS that provides the same glob ordering as the original study then the issue would have remained undiscovered.
|
# ? Oct 14, 2019 00:56 |
|
CarForumPoster posted:Yea, auto installing the latest python version for an environment of an otherwise known config seems like begging for problems. It's kind of a ghetto way to do version control but yeah, it makes sense. If you wrote software for Python 3.7, you don't want someone to assume that it'll continue working fine under Python 4.5 in the year 2057
|
# ? Oct 14, 2019 00:59 |
|
So I've worked with Python GUI development before and packaging is such a nightmare. https://build-system.fman.io/docs/ I just found this which lets you make a pyqt5 app and build it with `fbs freeze` or create an installer with `fbs installer` and ran through their little tutorial and everything worked flawlessly. Does anyone have any experience going beyond the tutorial with this thing? It seems way too good to be true.
|
# ? Oct 15, 2019 04:55 |
3.8 is here.
|
|
# ? Oct 15, 2019 10:09 |
|
cinci zoo sniper posted:3.8 is here. The f-string debug unpacking shorthand or whatever it is seems neat I guess
|
# ? Oct 15, 2019 17:24 |
|
The walrus operator looks neat but I'll have to practice with it to see where it's good and more importantly where it might impair readability.
|
# ? Oct 17, 2019 19:06 |
|
Yeah, looks possible to overdo but I'll probably end up using it.
|
# ? Oct 17, 2019 19:57 |
|
I've been playing around the multiprocessing library, and it looks like for what I was hoping to do with it (run a function which needs to execute on the order of 10,000 times, taking a total of roughly 1-2 seconds with a single thread), the overhead would use more time than I'd save. Are there any other solutions for parallel processing in Python under Windows 10, ones that don't add major overhead? This function is probably too complex to be a good candidate for CUDA.
|
# ? Oct 19, 2019 03:21 |
|
Is that 1-2 seconds overall, or 1-2 seconds per function call? If the former, you can probably still get a small performance increase but it may not be noticeable. If the latter, multiprocessing is extremely well-suited to your problem But... you should use concurrent.futures instead. It's basically a high-level wrapper for multiprocessing that was introduced in Python 3.2. It's multiprocessing (and also multithreading), but even easier to use. If you're mostly dealing with primitives and arrays, an even better and way-faster option is to use Numba to compile your function. Numba comes with a bunch of parallelization features, or you can just turn off the GIL and use your own multithreading with concurrent.futures if that's what you prefer.
|
# ? Oct 19, 2019 07:43 |
|
QuarkJets posted:Is that 1-2 seconds overall, or 1-2 seconds per function call? If the former, you can probably still get a small performance increase but it may not be noticeable. If the latter, multiprocessing is extremely well-suited to your problem It's 1-2 seconds overall. If concurrent.futures is based on the multiprocessing library, wouldn't it have the same problem with overhead as using multiprocessing by itself? It looks like Numba is my best option here (short of rewriting the function with Cython). Thank you.
|
# ? Oct 19, 2019 13:57 |
|
Found a list of some use cases for the new walrus operator. I'm sure going forward we're all going to hear a million times "what's it good for?". https://github.com/vlevieux/Walrus-Operator-Use-Cases
|
# ? Oct 19, 2019 23:04 |
|
Cockmaster posted:It's 1-2 seconds overall. If concurrent.futures is based on the multiprocessing library, wouldn't it have the same problem with overhead as using multiprocessing by itself? Yes, that's right. I'd advise using concurrency.futures if your problem was 1-2 seconds per function call * 10000 iterations, but it probably won't be good when each function call is 1-2 ms Also yeah, Numba is great and I prefer it over Cython, both for ease of use and for performance. In fact if you can compile the function in nopython mode then it may run so fast that you don't even need to consider parallelism
|
# ? Oct 19, 2019 23:19 |
|
Thermopyle posted:Found a list of some use cases for the new walrus operator. I'm sure going forward we're all going to hear a million times "what's it good for?". The list comprehension stuff seems genuinely good, but the rest is a bit meh.
|
# ? Oct 20, 2019 11:30 |
|
Private Speech posted:The list comprehension stuff seems genuinely good, but the rest is a bit meh. I'm sure that this is at least partially because I haven't really used the walrus operator yet and I'll need time to adjust, but it seems so unintuitive and makes the code way less readable for me.
|
# ? Oct 20, 2019 14:08 |
|
If I understand it properly, my main use will be shortening...code:
|
# ? Oct 20, 2019 14:49 |
Sad Panda posted:If I understand it properly, my main use will be shortening... This is my current takeaway as well, though I rarely use regex for anything so I'm sure the match shortcut might come in handy for those that do. Some other examples at the link Thermopyle shared might come in handy.
|
|
# ? Oct 20, 2019 15:34 |
|
punished milkman posted:I'm sure that this is at least partially because I haven't really used the walrus operator yet and I'll need time to adjust, but it seems so unintuitive and makes the code way less readable for me. I've used it in several languages and it's still unintuitive and hard to read. I think it's probably going to be over-used.
|
# ? Oct 20, 2019 17:53 |
|
There was some discussion a few pages back about the Obey the Testing Goat book for TDD with Python/Django. Just wanted to chime in and say I bought it because of whoever mentioned it and can vouch that it rules. If you're like me in that you've been doing lots of work with Django but have been neglecting doing adequate testing because it seems hard, overwhelming and maybe a waste of time, you should probably read this book.
|
# ? Oct 21, 2019 14:09 |
|
Is this an OK place to ask for help with something? I've started working with pandas dataframes/pivot tables at my job. I have a pivot table that looks like this: How do I add subtotals for elements of the middle level, e.g., a subtotal listing under "B" that sums over elements 2-14? I've found solutions online, but I haven't fully understood the reasoning, and trying to apply the solutions blindly has given me different kinds of unsatisfactory results.
|
# ? Oct 22, 2019 19:48 |
|
code:
One thing to get used to, if you're coming from Excel, is that you wouldn't put the subtotals "under" the raw data. I mean, you could probably do it, with the above line then assigning new rows, but it would be awkward. Better to use pandas to do the calculations, then if you need a pretty output at some point you would output to html or something.
|
# ? Oct 22, 2019 20:21 |
|
SurgicalOntologist posted:
Thanks for the reply. I suppose that (the prettiness) is the real question. For context, in this unit people have been taking raw data reports they get weekly, pasting them into a workbook, and constructing Excel pivot tables by hand for a whole host of tasks. The goal is to give them the same output they're used to programmatically.
|
# ? Oct 22, 2019 21:18 |
|
Multilevel indicies destroy my brain and I hates them. I like to convert stuff to tall format and operate that way, then pivot back to what form the output needs to be in. (I think that's called Tidy Data? Dunno)
|
# ? Oct 22, 2019 23:29 |
|
KICK BAMA KICK posted:Reread some code I wrote before finishing Obey the Testing Goat (again, strong recommend, not sure anything has ever helped me as much as that) and realizing I had actually rolled my own extremely stupid version of unittest.mock to fake an external API, return some fake data, capture the arguments used, the whole nine yards. jumping on the testing train myself now! any advice from windows vets on getting geckodriver/selenium working there? It seems annoying but I haven't messed with it too much yet Business fucked around with this message at 15:39 on Oct 23, 2019 |
# ? Oct 23, 2019 15:20 |
|
Business posted:jumping on the testing train myself now! any advice from windows vets on getting geckodriver/selenium working there? It seems annoying but I haven't messed with it too much yet
|
# ? Oct 23, 2019 15:36 |
|
Business posted:jumping on the testing train myself now! any advice from windows vets on getting geckodriver/selenium working there? It seems annoying but I haven't messed with it too much yet I haven't used it on Windows but maybe check out webdriver_manager and see if it works for you (https://github.com/SergeyPirogov/webdriver_manager). Installs via pip and automatically downloads the latest version of the needed driver for whatever browser you're working with.
|
# ? Oct 23, 2019 15:38 |
|
Business posted:jumping on the testing train myself now! any advice from windows vets on getting geckodriver/selenium working there? It seems annoying but I haven't messed with it too much yet You just download geckodriver and use it. Nothing special about windows.
|
# ? Oct 23, 2019 17:05 |
|
|
# ? May 27, 2024 02:43 |
|
Thanks yeah I was overthinking it. Messed something up the first try and based on google got the impression it was more complicated than it actually was
|
# ? Oct 23, 2019 17:39 |