|
Bourricot posted:I'm currently doing some Python for a personnal toy project. I'd like to use this as an opportunity to learn how to do things properly (re linting, auto-formatting...). So I'm asking for some goonpinions on the tools to do Python properly in 2021 (I'm already using mypy* and pylint): All of this is more structured than most python non-toy projects. venv and requirement.txt are fine. Two pitfalls I've hit with requirements multiple times, especially as projects get >2 years old, are these: 1. pip freeze as your requirements.txt tends to be asking for trouble. This is due to the install order of my main packages requirements conflicting. The most reliable way I've found is as I install into a clean venv, set the package I installed version in the order that I installed them in E.g. if I installed pandas first, the first line of the requirement.txt should be pandas==1.2.2 In general, if the order in my requirement.txt works locally and loosely matches the order I install them in, this has been reliable. 2. A few of of my deps will not freeze their versions. I recently had one that installed SQLAlchemy, and any installs of it after a certain version of SQLAlchemy fail. This actually is a case against #1 and suggests you SHOULD fix your versions of sub packages. I've however found it easier to fall in this pit once per year and have a shorter, readable requirements.txt. The solution was to install SQLAlchemy with a fixed version before the dep that didn't freeze it. You can of course also inspect your deps for whether they do this really dumb thing. I really like pycharm's built in highlighting and reformat options. EDIT: P.S. Someone please tell me the proper way to do this to mitigate both of those pitfalls.
|
# ? May 17, 2021 13:44 |
|
|
# ? May 28, 2024 14:38 |
|
In a shell I can run this snippet and it works:code:
|
# ? May 17, 2021 17:08 |
|
School of How posted:In a shell I can run this snippet and it works: It's an idiosyncrasy of python 3. https://stackoverflow.com/questions/1463306/how-does-exec-work-with-locals Now the real mystery is why it still works in the shell. Probably some bullshit.
|
# ? May 17, 2021 17:34 |
|
Bruegels Fuckbooks posted:It's an idiosyncrasy of python 3. I figured it out: code:
|
# ? May 17, 2021 17:43 |
|
Bourricot posted:I'm currently doing some Python for a personnal toy project. I'd like to use this as an opportunity to learn how to do things properly (re linting, auto-formatting...). So I'm asking for some goonpinions on the tools to do Python properly in 2021 (I'm already using mypy* and pylint): I tend to use black, because it is at least consistent even if sometimes things might initially look slightly odd. My only major problem is their stance on only supporting pyproject.toml AMD not offering support for setup.cfg, since that has an impact on packaging. Their defaults are fairly sensible, too.
|
# ? May 17, 2021 18:54 |
|
Im starting to think i'm falling a bit behind on my python knowledge. Having a browse of the module standard library and theres a *tonne* of stuff I dont know about in there. I mean, I've been using python since the 1990s back wthen it was the 1.x range , but I'm still coming to terms with putting type annotations in things (good idea, btw and well overdue) but uh dataclasses and stuff. Guess I got some catching up to do... And this stuffs been in there fore years too. Goddamn it I'm feeling old.
|
# ? May 21, 2021 17:39 |
|
So I was helping a student fix a project and ran into a really confusing bug that I went in circles trying to figure out. It's solved, but now I come to the thread to ask this: Is there any valid reason to put something like a @classmethod decorator on a class's __init__ method? Or any decorator, really? I really thought the interpreter would have had some special stuff in it for making sure those methods didn't get decorators thrown on them.
|
# ? May 24, 2021 21:41 |
Epsilon Plus posted:So I was helping a student fix a project and ran into a really confusing bug that I went in circles trying to figure out. It's solved, but now I come to the thread to ask this: Seems like you could conceivably want a logging decorator or something similar on your init?
|
|
# ? May 24, 2021 21:57 |
|
That makes some sense, thank you.
|
# ? May 24, 2021 22:06 |
|
I think there might be legitimate use cases for a decorator on __init__, but I can’t think of any good reason to use @classmethod there.
|
# ? May 24, 2021 22:06 |
|
There can be all kinds of good reasons to stick a decorator onto __init__. Decorators are very flexible, and if properly implemented an __init__ decorator should pose no problems I can't think of any specific scenario where you'd want to use @classmethod with __init__. It should work, but it's kind of weird and I think a code smell. Using @classmethod for what it's normally used for on __init__ could easily create big problems (for instance, if you tried to turn __init__ into a factory method) Like for instance this works fine and behaves as-expected Python code:
|
# ? May 24, 2021 22:07 |
|
Hoping someone can help me with this because I’m out of my element and pretty stuck. I need to make a subprocess calling function that has a few properties. 1) it needs to stream both stdout and stderr to the console. Using Popen.communicate() seems to cause the outputs to block til the process is done if I redirect stdout and stderr to subprocess.PIPE. Using stdout=None and stderr=None with Popen.communicate() seems to stream to the console, so that’s good, but that leads me to… 2) I need to capture the contents of stdout and stderr in their own text files, preferably writing to them as the process continues. Any idea how I do this? Sorry if it’s unclear I’m phone posting and also dumb
|
# ? Jun 7, 2021 17:20 |
|
punished milkman posted:Hoping someone can help me with this because I’m out of my element and pretty stuck. Can you just not stream to stdout and instead stream to a file/logger and read the log file with something like notepad++ which will stream updates to files?
|
# ? Jun 7, 2021 17:45 |
|
I think .communicate waits for the process to exit before returning? I think you can do something likecode:
|
# ? Jun 7, 2021 18:24 |
|
You can directly pass an open file to the stdin/stderr args of Popen any of the subprocess helper functions. Edit: oops, misunderstood. I missed the console part. You want to do a tee basically. I don't know if there's any shortcut besides iterating. Well, maybe just use tee outside python. SurgicalOntologist fucked around with this message at 19:23 on Jun 7, 2021 |
# ? Jun 7, 2021 19:15 |
|
Can someone help me with a stupid boto3 issue I'm dealing with? I have this code block using boto3's route 53 library to update a dns record in awscode:
code:
Edit: Very dumb typo... 'Action': 'CREATE': to 'Action': 'CREATE', fixed it. post hole digger fucked around with this message at 20:42 on Jun 7, 2021 |
# ? Jun 7, 2021 20:12 |
|
dict literal keys must be quoted in python oh wait that's kwargs. I have no idea.
|
# ? Jun 7, 2021 20:42 |
|
Nevermind
|
# ? Jun 7, 2021 20:45 |
Stupid parser tricks When this happens it's funny to look at how it ended up with that unhelpful error message and try to figure out what it thought you were trying to pull off
|
|
# ? Jun 7, 2021 21:25 |
|
punished milkman posted:Hoping someone can help me with this because I’m out of my element and pretty stuck. Other posts have the right of it, I just wanted to add some more specific notes: this is made a little complicated by wanting to write the streams to separate files while printing both to the console. If you just wanted to do one or the other it'd be pretty easy. I know how to get you most of the way there, at least: create a Popen with stderr piped to stdout (using subprocess.STDOUT as the argment for stderr, I believe) and then iterate over Popen.stdout. You can write lines wherever you want, iteration won't complete until the process exits, you can print to the terminal, the issue with this scheme is that stderr and stdout are combined. I guess you could use PIPE for both stdout and stderr, then create a while loop that 1) does something with contents of stdout, 2) does something with contents of stderr, 3) repeat until both buffers are empty and poll() indicates that the process terminated. I've never tried doing something like this, and there may be a better way.
|
# ? Jun 8, 2021 00:14 |
I have a Flask/SQLAlchemy question that I'm having trouble figuring out the right combination of words to even Google to get help. I'm making a simple message board app for a class project and am trying to work out a query that will give me the following for a page of threads: - The threads themselves. - The number of posts in each thread. - The created_at datetime of the most recent post in each thread. - The username of the people who made those most recent posts. I've managed to get the first three with this: code:
|
|
# ? Jun 8, 2021 02:05 |
|
QuarkJets posted:Other posts have the right of it, I just wanted to add some more specific notes: this is made a little complicated by wanting to write the streams to separate files while printing both to the console. If you just wanted to do one or the other it'd be pretty easy. Yep this is more or less what I landed on. I toyed around with some asyncio and aiofile stuff for a bit and quickly got overwhelmed and moved back to some synchronous code with some success. As thanks to everyone, here’s a lovely photo of my code
|
# ? Jun 8, 2021 04:31 |
|
punished milkman posted:a lovely photo of code this is haram
|
# ? Jun 8, 2021 05:01 |
|
wouldn't it be easier to just copy-paste the code?
|
# ? Jun 9, 2021 02:56 |
Tell that to the guys on my team who just relentlessly spam intricately marked-up screenshots faster than they can type apparently
|
|
# ? Jun 9, 2021 03:10 |
|
Data Graham posted:Tell that to the guys on my team who just relentlessly spam intricately marked-up screenshots faster than they can type apparently I can’t imagine why anyone would ever need to screenshot code Like…every communication platform supports code markdown blocks
|
# ? Jun 9, 2021 04:16 |
|
CarForumPoster posted:I can’t imagine why anyone would ever need to screenshot code Screenshots are old news. I take a scrolling video with my phone using full filters so that nobody can steal my awesome code. You might laugh but do you know how many people copy my code? That's right, none
|
# ? Jun 9, 2021 07:43 |
|
qsvui posted:wouldn't it be easier to just copy-paste the code? I guess not everyone wants to log into their somethingawful.com account on their work computer
|
# ? Jun 9, 2021 13:11 |
CarForumPoster posted:I can’t imagine why anyone would ever need to screenshot code I've asked them to stop and they won't stop I guess maybe it's because they can circle things in red and make big red arrows pointing to things and highlight stuff in yellow which is great for making me feel like an absolute dullard who needs to be communicated with via flashcards
|
|
# ? Jun 9, 2021 13:17 |
HappyHippo posted:I guess not everyone wants to log into their somethingawful.com account on their work computer So use one of the several hundred pastebins or online repls
|
|
# ? Jun 9, 2021 13:56 |
|
Data Graham posted:I've asked them to stop and they won't stop Context is important here too, if they're doing this not to share the code with you but to point something out in something you already have it makes sense. I send my colleagues marked up screenshots of config files or web interfaces all the time, but in the context of "this setting is here" or "holy poo poo look what I just found in $newcustomer's equipment that we'll need to unfuck". If I'm actually sharing code with them I use a code block as any reasonable person should.
|
# ? Jun 9, 2021 16:23 |
|
Bundy posted:So use one of the several hundred pastebins or online repls taking a terrible photo of your computer screen is much funnier
|
# ? Jun 9, 2021 16:24 |
|
Bundy posted:So use one of the several hundred pastebins or online repls I would be hesitant to do that for any corporate codebase that's not open source
|
# ? Jun 9, 2021 17:59 |
|
QuarkJets posted:I would be hesitant to do that for any corporate codebase that's not open source I wouldn't put anything on SomethingAwful that I wouldn't put on Pastebin. At least with Pastebin it isn't automatically indexed if you set it to unlisted.
|
# ? Jun 9, 2021 18:09 |
|
Is there a way to use subprocess for internal functions? Or, a way to set timeout value to a function that's fairly simple (in the standard library) maybe with multiprocessing module? I'm running into a fun issue where DNS lookups for .local are ignoring socket.timeout value on Mac (probably due to bonjour integration / special handling), and the default timeout is 5 seconds. This is an automated test that hits failure case if I forgot to connect to VPN so I just want it to use a very short timeout instead of sitting there for 5 seconds and then failing. As a workaround I've put a function that returns the value of socket.gethostbyname() into a separate file and the very first test is now a fixture that calls subprocess.check_output() with timeout=1 parameter on that file. Works perfectly, saving me four seconds. Amazing. Just wondering if there's a way to make subprocess/standard library do this with local functions as opposed to calling separate python shell command.
|
# ? Jun 9, 2021 21:42 |
|
mr_package posted:Is there a way to use subprocess for internal functions? Or, a way to set timeout value to a function that's fairly simple (in the standard library) maybe with multiprocessing module? I'm running into a fun issue where DNS lookups for .local are ignoring socket.timeout value on Mac (probably due to bonjour integration / special handling), and the default timeout is 5 seconds. This is an automated test that hits failure case if I forgot to connect to VPN so I just want it to use a very short timeout instead of sitting there for 5 seconds and then failing. You can create a multiprocessing.Process with the target= argument as the callable you want to run, ideally using a Pipe or Queue to feed your data back to the main process. You then call .start() and then .join(timeout) with whatever timeout you want before checking the Pipe/Queue and calling it a day. If your underlying callable releases GIL you can even do this with threading.Thread instead.
|
# ? Jun 9, 2021 22:56 |
|
concurrent.futures is the sexy new(er) wrapper for multiprocessing and makes this sort of stuff pretty easy. The Future object that gets returned has a wait method with an optional timeout argument
|
# ? Jun 10, 2021 02:07 |
|
OnceIWasAnOstrich posted:You can create a multiprocessing.Process with the target= argument as the callable you want to run, ideally using a Pipe or Queue to feed your data back to the main process. You then call .start() and then .join(timeout) with whatever timeout you want before checking the Pipe/Queue and calling it a day. If your underlying callable releases GIL you can even do this with threading.Thread instead. It's default behavior is to fork() to make a copy of the process, then keep using it without calling exec(). This has always been forbidden by POSIX, but is commonly done. It will generally work fine as long as absolutely everything in the process holds no locks at the moment the fork occurs, which you have no way of assuring that is true in general (you can't easily prove that some library doesn't make a thread that it uses internally)
|
# ? Jun 10, 2021 04:18 |
|
I did some experiements with ThreadPoolExecutor and ProcessPoolExecutor. A few surprises, such as the behaviors of context managers, which wait for everything to finish before exiting. This is probably a good thing, I was just surprised that getting future.result(timeout=1) did not have any effect e.g. that raises TimeoutError, but does not __exit__ the context manager. I was also surprised that you basically cannot easily kill threads/processes that you launch. There are third-party libraries and what appear-- to me-- to be very hack-y approaches to dealing with this to be found on SO. Perhaps they are fine, perhaps this is just the way it is done. This is my first look at concurrency, do other languages handle this pretty much the same way or is it Python-specific? I'm thinking for example if you were writing something in go or rust or C# or whatever and you wanted to terminate a process early, is it easier? And if so, what become the best way to do this in Python? It seems like I find myself back at square one: how do you gracefully __exit__ from a method if it is taking too long? In the current code, you get error messages when you expect to have them, but the actual program does not exit until everything is finished. For a 5 seconds DNS timeout, it's liveable but what if there's something longer that you need to kill if it exceeds 60s, or 300s, or whatever? code:
|
# ? Jun 10, 2021 21:17 |
|
|
# ? May 28, 2024 14:38 |
|
You can kill a running process by just calling cancel() on its Future. Tested code:Python code:
|
# ? Jun 11, 2021 03:07 |