|
That doesn't actually kill the process, it just stops waiting for the result:Python code:
code:
mr_package posted:I was also surprised that you basically cannot easily kill threads/processes that you launch. There are third-party libraries and what appear-- to me-- to be very hack-y approaches to dealing with this to be found on SO. Perhaps they are fine, perhaps this is just the way it is done. This is my first look at concurrency, do other languages handle this pretty much the same way or is it Python-specific? I'm thinking for example if you were writing something in go or rust or C# or whatever and you wanted to terminate a process early, is it easier? It is possible to safely abruptly kill a process if you have appropriate permissions, but it's not something I would expect a processing pool library to make easy. Again, your code should have a way to gracefully cancel (returning the worker to the pool), not forcefully rip down a piece of the pool infrastructure and hope the pool implementation heals.
|
# ? Jun 11, 2021 03:43 |
|
|
# ? May 28, 2024 15:21 |
|
What does the correct way to do this look like? I have a piece of code I know might take too long. How do I set up the call to it so that it terminates in an appropriate time? This is not a standard thing? Or, is it that the core issue is socket.gethostbyname() has a 5s timeout and I need to respect that? In other words, shoehorning control of that process into a MP library is the wrong approach: I should not use socket.gethostbyname() at all if the timeout behavior is wrong for my use case.. For example, I could write my own (what could go wrong?) or use another library such as https://www.dnspython.org/ that has a timeout and kills itself rather than my trying to kill it if it takes too long. Is that the way this type of thing should be done? As an aside, does anyone know offhand what happens when you call subprocess.run() with a timeout? Does the process continue to run in the background and then exit normally however long it takes? I'm wondering if the caller just ignores it and moves on or if it actually communicates to the process that it should quit/cancel (presumably the process would be responsible for then handling this in a mostly graceful way).
|
# ? Jun 11, 2021 06:42 |
|
mr_package posted:What does the correct way to do this look like? I have a piece of code I know might take too long. How do I set up the call to it so that it terminates in an appropriate time? This is not a standard thing? For lots of stuff that wasn't ever intended to be used asynchronously, there won't be a way to do that. gethostbyname() doesn't have a way to cancel it. What you can do is make the code that launched it not wait anymore & make it so that once the slow call does return, your code ignores the result and just exits: Python code:
mr_package posted:As an aside, does anyone know offhand what happens when you call subprocess.run() with a timeout? Does the process continue to run in the background and then exit normally however long it takes? I'm wondering if the caller just ignores it and moves on or if it actually communicates to the process that it should quit/cancel (presumably the process would be responsible for then handling this in a mostly graceful way).
|
# ? Jun 12, 2021 01:38 |
Hey yo. What's the least stupid way to do the following: * Define a "templated" arithmetic function in a string, for example "0.5 * {width}" * Store that string in a database (i.e. a Django CharField) * At rendering time, eval the string against a set of kwargs and spit out a result I know I can just like eval(templated_string.format(**kwargs)), but that seems dangerous even if I own the data flow end-to-end. Is there a good way to do this that avoids eval()?
|
|
# ? Jun 13, 2021 00:49 |
|
I think eval() is probably the least stupid way to do it, you just need to do some text validation on the input to ensure that you're seeing arithmetic and not something else. Any non-arithmetic characters should raise an exception prior to running with eval()
|
# ? Jun 13, 2021 01:05 |
|
The old Armin Ronacher article about security concerns with user provided format strings might also be worth a read. On mobile but should be easy to Google.
|
# ? Jun 13, 2021 02:19 |
|
It's probably overkill and I have no idea what it offers for serializing expressions, but it could be interesting to use the symbolic computation library SymPy to represent variables and expressions. I think it can parse strings so you wouldn't have to worry about that. You probably don't need much math stuff but you can evaluate expressions, and maybe some other features like simplification or equation display would be appealing.
|
# ? Jun 13, 2021 08:20 |
|
Data Graham posted:Hey yo. What's the least stupid way to do the following: Probably a stupid idea: what about storing the arithmetic function as a serialized object in the database? So instead of a storing the string "0.5 * {width}", you could serialize the function "lambda width: 0.5 * float(width)" into a string and store that. Afterwards you can retrieve the string, deserialize it into a function and call it. See this SO for ways to do that: https://stackoverflow.com/questions/1253528/is-there-an-easy-way-to-pickle-a-python-function-or-otherwise-serialize-its-cod Note: I don't think this is actually that much safer than your version if you're worried about people modifying the database (on the contrary). But it should be more safe if you're worried about code injection via the width parameter. edit: actually, perhaps storing "lambda width: 0.5 * float(width)" instead of "0.5 * {width}" might be worth considering as well? Obviously more complicated, but python will complain if "width" is not parseable to a float. Walh Hara fucked around with this message at 09:37 on Jun 13, 2021 |
# ? Jun 13, 2021 09:24 |
Good feedback, thanks all. I guess I'll just preface the eval() with some regex checking, since these are pretty simple expressions I'm working with. Maybe a restrictive length limit too in case someone is somehow able to fit "import User" and "User.objects.all().delete()" into it lol I mainly wanted to see if there was a commonly-used best-practices approach for this sort of thing that I didn't know about.
|
|
# ? Jun 13, 2021 12:34 |
|
I mean unfortunately I think the sort of real answer is "if it matters, don't do that". Like, if this is a toy project, whatever, string.format() is probably fine. But if it's a real project that's, for instance, on the internet, you probably don't want to use strings for it at all. You want to store a computation tree, like an AST (and maybe literally an ast.AST instance that you poke the user-supplied operands into), after parsing and sanitizing user input at the point of entry, both for security reasons and for error locality reasons.
|
# ? Jun 13, 2021 13:26 |
|
I'm having a hell of a time trying to figure out how to parse this into a dictionary. https://forecast.weather.gov/MapClick.php?lat=41.4273&lon=-90.0224&unit=0&lg=english&FcstType=dwml
|
# ? Jun 17, 2021 00:31 |
|
Why not use their API? https://api.weather.gov/points/41.4273,-90.0224 https://weather-gov.github.io/api/general-faqs
|
# ? Jun 17, 2021 01:58 |
|
sleppy posted:Why not use their API? I'd love to butt unfortunately it has severe caching issues that aren't going to be resolved any time soon. Kind of a big deal for time sensitive information. https://github.com/weather-gov/api/discussions/224
|
# ? Jun 17, 2021 02:54 |
|
I'm having an issue trying to add a matplotlib graph to a tkinter GUI where the legend to the graph is getting cut off if I move it to below the chart. Does anyone know a way to display the legend when it is outside of the chart? FigureCanvasTkAgg doesn't have a height argument so I can't just stretch the viewable area. It looks like it is some kind of automatic resizing problem that I can't figure out a way around. Here's the test code: Python code:
|
# ? Jun 17, 2021 14:42 |
SirPablo posted:I'd love to butt unfortunately it has severe caching issues that aren't going to be resolved any time soon. Kind of a big deal for time sensitive information. Gross. I would look at https://docs.python.org/3/library/xml.etree.elementtree.html and adapt this https://stackoverflow.com/questions/2148119/how-to-convert-an-xml-string-to-a-dictionary
|
|
# ? Jun 17, 2021 14:49 |
|
Thanks will give that a try.
|
# ? Jun 17, 2021 14:52 |
|
Deadite posted:I'm having an issue trying to add a matplotlib graph to a tkinter GUI where the legend to the graph is getting cut off if I move it to below the chart. I've never had this problem in the context of another GUI but I've definitely run into similar issues with bits of a matplotlib figure getting rendered outside the bounds of an image. It is usually something that a call to tight_layout() or other layout-modifying functions can address.
|
# ? Jun 17, 2021 14:57 |
|
Have I misunderstood pip’s version specifiers, or is it doing something weird here? If I run code:
code:
(I originally found this problem with a requirements.txt file that had a bunch of >= dependencies, also this is installing from an Azure artefacts repository that’s mirroring PyPI if that makes any difference)
|
# ? Jun 17, 2021 15:05 |
|
I got myself familiar with xpath once, and now it's the single hammer I use for every single XML parsing problem I encounter. Xml.etree already has a decent support for the syntax.
|
# ? Jun 17, 2021 15:13 |
|
DoctorTristan posted:I don’t have the cryptography package installed, so can’t see any conflict here? Unless it’s trying to install all four versions and finding a conflict? Is it possible that something else in the requirements file has a cryptography dependency? I think pipdeptree will let you know.
|
# ? Jun 17, 2021 15:22 |
|
DoctorTristan posted:Have I misunderstood pip’s version specifiers, or is it doing something weird here? No, you are using it right. Something in the dependency resolution it is doing seems to be calling for cryptography. I would normally say that something else you have installed or are installing has a conflict with the cryptography version number. What version of Python are you using? Maybe there are no cryptography packages for your Python in that range or the repo/your environment is busted. You can also try pip installing a version of the package in the appropriate version range and see what it does. Now that I think about it if there was an actual conflicting dependency pip would normally list it in that same output. OnceIWasAnOstrich fucked around with this message at 15:28 on Jun 17, 2021 |
# ? Jun 17, 2021 15:23 |
|
OnceIWasAnOstrich posted:No, you are using it right. Something in the dependency resolution it is doing seems to be calling for cryptography. I would normally say that something else you have installed or are installing has a conflict with the cryptography version number. What version of Python are you using? Maybe there are no cryptography packages for your Python in that range or the repo/your environment is busted. Should have thought of doing that that myself, but tried it and found that the underlying problem is that there’s no cryptography package on this piece of poo poo azdo mirror of ours, so none of the packages can install. No idea why pip wouldn’t just say that instead of bullshitting about conflicts, but hey ho. In conclusion: gently caress computers
|
# ? Jun 17, 2021 16:16 |
|
Isn’t cryptography the one that switched to linking against rusttls recently? May not be a big deal depending on your environment but something to keep in mind. Also, agreed re: computers.
|
# ? Jun 17, 2021 16:31 |
|
OnceIWasAnOstrich posted:I've never had this problem in the context of another GUI but I've definitely run into similar issues with bits of a matplotlib figure getting rendered outside the bounds of an image. It is usually something that a call to tight_layout() or other layout-modifying functions can address. Thanks, tight_layout is exactly what I was looking for.
|
# ? Jun 17, 2021 16:49 |
|
Hed posted:Isn’t cryptography the one that switched to linking against rusttls recently? Possibly true - I don’t know much about it other than that msal depends on it (msal is the Microsoft authentication library - employer uses OAuth all over the shop so I have to use this package a lot) I also found that if I try to do this with two packages with missing dependencies I can send pip into an infinite loop, which is really stupid.
|
# ? Jun 18, 2021 10:58 |
|
nvm
punk rebel ecks fucked around with this message at 02:05 on Jul 11, 2021 |
# ? Jul 11, 2021 01:31 |
|
Pandas question: I am trying to make row wise exponential regression in order to fill in NaN values in a dataframe. Size of data is only 16K rows x 7 cols, so this will work fine but I get a twinge when I iterate over a pandas object in a function I'm going to apply. Whats the right way to accomplish this? Example data: code:
I made the below code which returns a list with all the values filled in, note the for loop. This "feels bad" but I can't think of a better way. code:
|
# ? Jul 13, 2021 13:53 |
|
Does DataFrame.interpolate() do what you need ?
|
# ? Jul 13, 2021 14:22 |
|
DoctorTristan posted:Does DataFrame.interpolate() do what you need ? fuckin probably I didnt see an exponential curve at first glance but I bet one of those will work. At a glance it looks like it wouldnt want to fill backward, but I prob just need to study docs a bit more. This data never dips below 0 and is always exponentially increasing to the right. I always tell people "someone already probably did it better" when they have a python usecase that seems like it should be common. Didnt follow my own advice. Thanks!
|
# ? Jul 13, 2021 14:39 |
|
DoctorTristan posted:Does DataFrame.interpolate() do what you need ? TY again. Hilariously easy and fast solution. Take the log of the df, interpolate linearly across row, take exp of df. code:
|
# ? Jul 13, 2021 15:15 |
|
I'm writing my first python app that other people will use instead of just for my own purposes. How do I set things up for users to install the dependencies and run it? Is it just put in the readme: code:
|
# ? Jul 13, 2021 21:52 |
|
It depends on who you're targeting, you can freeze python applications into executables or just have them run on a webserver if you really want to make it easy. For someone cloning your repo, those instructions are sufficient, publishing to PyPi would be better though
|
# ? Jul 13, 2021 22:37 |
|
I'm pretty new to python and I've hit a major stumbling block in trying to test out an old github repo. It has a requirements.txt with a good 7-8 libraries listed with specific versions, and I've inferred that this was developed on python 3.5. I've tried creating a venv with my system 3.7 python and using pip and it dies trying to install one of the libs. I tried using a conda environment with all sorts of python versions from 3.4-3.9 -- thinking maybe conda is smarter about this than pip -- and it dies on a different package with each version. There doesn't seem to be a way to get all the requirements to install. I've managed to get slightly different versions of libs to install except for matplotlib and pyclipper now, and I'm wondering if there's some sort of standard way of working this out and maybe I'm just making this hard for no reason? I've tried having PyCharm process the requirements.txt with conda and even manually just installing the packages. Sorry, don't want to get all stackoverflow on the thread, just wondering if there's some obvious thing I should be using for this type of problem. The whole world of pip, conda, miniconda, anaconda has me reaching all over.
|
# ? Jul 15, 2021 03:01 |
|
former glory posted:I'm pretty new to python and I've hit a major stumbling block in trying to test out an old github repo. It has a requirements.txt with a good 7-8 libraries listed with specific versions, and I've inferred that this was developed on python 3.5. I've tried creating a venv with my system 3.7 python and using pip and it dies trying to install one of the libs. Honestly it kind of sounds to me that time has ended up making that collection of libraries at the versions frozen in the requirements mutually incompatible through transitive dependency, like maybe - app requires a at version 1 and b at version 2 - a at version 1 requires c at version > 3 - c now has a version 4, which matches a's requirement, but in turn requires, say, b at version 3 - oops you're broken I don't think there's a specific tool that can base its work on requirements that can help, the way around this is what poetry and pipenv do which is to take something that has these semantic requirements like a requirements.txt and freeze them into a checked-in exact version snapshot of the world (or at least the venv) which is what you later use to reinstall things, and you can periodically update that snapshot to keep track of updates to dependencies and transitive dependencies. The path forward might be to scrub through the requirements and the pip or conda output and see which dependencies are causing the problem and add or edit restrictions until you can massage the dependency tree back into a working state.
|
# ? Jul 15, 2021 12:32 |
|
former glory posted:I'm pretty new to python and I've hit a major stumbling block in trying to test out an old github repo. It has a requirements.txt with a good 7-8 libraries listed with specific versions, and I've inferred that this was developed on python 3.5. I've tried creating a venv with my system 3.7 python and using pip and it dies trying to install one of the libs. Phobeste explained it well. Here's how I'd approach the problem. This has worked for me in under an hour but there might be a better way. If you're installing stuff that needs binaries like VC++, CUDA for deep learning, selenium for web scraping, or GraphViz for ML/network visualizations I highly recommend you conda install as much as possible. Google for the package to find it on conda-forge. Step 1) If the code was written for Python 3.5, which is quite old by now and missing some useful stuff, you should still install that unless you have some other requirements that can't be met by Python 3.5 (e.g. for deploying a web app) Step 2) I'd look at the requirements.txt, is it in alphabetical order meaning someone made it with pip freeze? 2a) If yes, pip install whatever is imported in the various .py files at the version frozen in requirements.txt. You dont have to have perfect coverage, you just want to install first whatever the authors probably installed first. THEN install requirements.txt, comment out things it dies on by adding a # in front of them in requirements.txt until the install is successful. 2b) If no, install requirements.txt and see where it dies, the comment out things it dies on by adding a # in front of them in requirements.txt until the install is successful. 3) At the very end, manually install the commented out ones, sleuthing on PyPi's history for dates to see what versions of it and its dependencies were published when. If all else fails, install the latest or a later version of the commented out thing with pip or conda. This only works for smallish packages where you can reasonably test for unexpected behavior, as any of those packages and any of their deps may have had API changes.
|
# ? Jul 15, 2021 14:34 |
|
Thanks, both posts were really helpful. I'm going to attack it again later tonight with that methodology. I'll keep a snapshot of my entire venv once I finish up my current and first big python project to hopefully avoid this when I inevitably unbox it a while from now.
|
# ? Jul 15, 2021 20:22 |
|
former glory posted:Thanks, both posts were really helpful. I'm going to attack it again later tonight with that methodology. I'll keep a snapshot of my entire venv once I finish up my current and first big python project to hopefully avoid this when I inevitably unbox it a while from now. If you get stuck, post the requirements.txt and I'll take a stab at it (no promises), assuming "a good 7-8 libraries listed with specific versions" means that's the total number of requirements (as opposed to "7-8 of the 75 dependencies have versions pinned").
|
# ? Jul 16, 2021 01:07 |
|
QuarkJets posted:It depends on who you're targeting, you can freeze python applications into executables or just have them run on a webserver if you really want to make it easy. For someone cloning your repo, those instructions are sufficient, publishing to PyPi would be better though Thanks. It's being written as an internal tool but my org decided to open source it once we're done. We're not really interested in hosting it, rather just letting other IT people pull it if they have the same niche use case. Your comment about publishing to PyPi caused me to learn about setup.py which I think is the answer I didn't know how to ask the right question about.
|
# ? Jul 16, 2021 02:09 |
|
Soylent Pudding posted:Thanks. It's being written as an internal tool but my org decided to open source it once we're done. We're not really interested in hosting it, rather just letting other IT people pull it if they have the same niche use case. This post summarizes 90% of PyPi repos last updated before Jan 2021
|
# ? Jul 16, 2021 03:01 |
|
|
# ? May 28, 2024 15:21 |
|
I am a coding newbie working through Learn Python The Hard Way. For exercise 36 he says to make a text-based adventure game. I decided to add quicktime events, which requires timed input. I found a complicated, Windows-only method on reddit and it works fine, but I decided to try to make the seemingly simple crossplatform solution work just for it's own sake. Here is what I have:code:
Is there some easy way to do what I want or should I stick to my Windows-only method?
|
# ? Jul 19, 2021 02:27 |