Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Bourricot posted:

I'm currently doing some Python for a personnal toy project. I'd like to use this as an opportunity to learn how to do things properly (re linting, auto-formatting...). So I'm asking for some goonpinions on the tools to do Python properly in 2021 (I'm already using mypy* and pylint):
- What is the recommended choice for code formatting: black? flake8? something else?
- Since it's going to be a basic script, I figure venv and a requirements.txt are good enough for my usecase. Am I mistaken?
- Any other pitfalls I might not be aware of?

*I know Python works well without static typing, but I wanted to give it a try.

All of this is more structured than most python non-toy projects. venv and requirement.txt are fine.

Two pitfalls I've hit with requirements multiple times, especially as projects get >2 years old, are these:
1. pip freeze as your requirements.txt tends to be asking for trouble. This is due to the install order of my main packages requirements conflicting. The most reliable way I've found is as I install into a clean venv, set the package I installed version in the order that I installed them in E.g. if I installed pandas first, the first line of the requirement.txt should be pandas==1.2.2 In general, if the order in my requirement.txt works locally and loosely matches the order I install them in, this has been reliable.

2. A few of of my deps will not freeze their versions. I recently had one that installed SQLAlchemy, and any installs of it after a certain version of SQLAlchemy fail. This actually is a case against #1 and suggests you SHOULD fix your versions of sub packages. I've however found it easier to fall in this pit once per year and have a shorter, readable requirements.txt. The solution was to install SQLAlchemy with a fixed version before the dep that didn't freeze it. You can of course also inspect your deps for whether they do this really dumb thing.

I really like pycharm's built in highlighting and reformat options.

EDIT: P.S. Someone please tell me the proper way to do this to mitigate both of those pitfalls.

# ? May 17, 2021 13:44

Adbot: ADBOT LOVES YOU

# ? May 26, 2024 09:53

School of How: Jul 6, 2013; quite frankly I don't believe this talk about the market

In a shell I can run this snippet and it works:

code:

>>> local_value = None
>>> exec("local_value = 4")
>>> local_value
4

exec modifies the value of a local variable. But when I try doing this in a unit test, it just doesn't work. The value of local_value is just None. Why is this? I'm using python 3.6.

# ? May 17, 2021 17:08

Bruegels Fuckbooks: Sep 14, 2004; Now, listen - I know the two of you are very different from each other in a lot of ways, but you have to understand that as far as Grandpa's concerned, you're both pieces of shit! Yeah. I can prove it mathematically.

School of How posted:

In a shell I can run this snippet and it works:
code:
>>> local_value = None
>>> exec("local_value = 4")
>>> local_value
4
exec modifies the value of a local variable. But when I try doing this in a unit test, it just doesn't work. The value of local_value is just None. Why is this? I'm using python 3.6.

It's an idiosyncrasy of python 3.
https://stackoverflow.com/questions/1463306/how-does-exec-work-with-locals

Now the real mystery is why it still works in the shell. Probably some bullshit.

# ? May 17, 2021 17:34

School of How: Jul 6, 2013; quite frankly I don't believe this talk about the market

Bruegels Fuckbooks posted:

It's an idiosyncrasy of python 3.
https://stackoverflow.com/questions/1463306/how-does-exec-work-with-locals

Now the real mystery is why it still works in the shell. Probably some bullshit.

I figured it out:

code:

g = {'local_value': None}
exec("local_value = 4", g)
g['local_value']

now it works, whatever.

# ? May 17, 2021 17:43

Hollow Talk: Feb 2, 2014

Bourricot posted:

I'm currently doing some Python for a personnal toy project. I'd like to use this as an opportunity to learn how to do things properly (re linting, auto-formatting...). So I'm asking for some goonpinions on the tools to do Python properly in 2021 (I'm already using mypy* and pylint):
- What is the recommended choice for code formatting: black? flake8? something else?
- Since it's going to be a basic script, I figure venv and a requirements.txt are good enough for my usecase. Am I mistaken?
- Any other pitfalls I might not be aware of?

*I know Python works well without static typing, but I wanted to give it a try.

I tend to use black, because it is at least consistent even if sometimes things might initially look slightly odd. My only major problem is their stance on only supporting pyproject.toml AMD not offering support for setup.cfg, since that has an impact on packaging. Their defaults are fairly sensible, too.

# ? May 17, 2021 18:54

duck monster: Dec 15, 2004

Im starting to think i'm falling a bit behind on my python knowledge. Having a browse of the module standard library and theres a *tonne* of stuff I dont know about in there. I mean, I've been using python since the 1990s back wthen it was the 1.x range , but I'm still coming to terms with putting type annotations in things (good idea, btw and well overdue) but uh dataclasses and stuff. Guess I got some catching up to do... And this stuffs been in there fore years too.

Goddamn it I'm feeling old.

# ? May 21, 2021 17:39

death cob for cutie: Dec 30, 2006; dwarves won't delve no more
too much splatting down on Zot:4

So I was helping a student fix a project and ran into a really confusing bug that I went in circles trying to figure out. It's solved, but now I come to the thread to ask this:

Is there any valid reason to put something like a @classmethod decorator on a class's __init__ method? Or any decorator, really? I really thought the interpreter would have had some special stuff in it for making sure those methods didn't get decorators thrown on them.

# ? May 24, 2021 21:41

a foolish pianist: May 6, 2007; (bi)cyclic mutation

Epsilon Plus posted:

So I was helping a student fix a project and ran into a really confusing bug that I went in circles trying to figure out. It's solved, but now I come to the thread to ask this:

Is there any valid reason to put something like a @classmethod decorator on a class's __init__ method? Or any decorator, really? I really thought the interpreter would have had some special stuff in it for making sure those methods didn't get decorators thrown on them.

Seems like you could conceivably want a logging decorator or something similar on your init?

# ? May 24, 2021 21:57

death cob for cutie: Dec 30, 2006; dwarves won't delve no more
too much splatting down on Zot:4

That makes some sense, thank you.

# ? May 24, 2021 22:06

susan b buffering: Nov 14, 2016

I think there might be legitimate use cases for a decorator on __init__, but I can�t think of any good reason to use @classmethod there.

# ? May 24, 2021 22:06

QuarkJets: Sep 8, 2008

There can be all kinds of good reasons to stick a decorator onto __init__. Decorators are very flexible, and if properly implemented an __init__ decorator should pose no problems

I can't think of any specific scenario where you'd want to use @classmethod with __init__. It should work, but it's kind of weird and I think a code smell. Using @classmethod for what it's normally used for on __init__ could easily create big problems (for instance, if you tried to turn __init__ into a factory method)

Like for instance this works fine and behaves as-expected

Python code:

class MyClass:
    @classmethod
    def __init__(cls, a):
        cls.a = a
        
x = MyClass(4)
print(x.a)  # prints 4

y = MyClass(5)
print(y.a)  # prints 5
print(x.a)  # prints 5

# ? May 24, 2021 22:07

punished milkman: Dec 5, 2018; would have won

Hoping someone can help me with this because I�m out of my element and pretty stuck.

I need to make a subprocess calling function that has a few properties.

1) it needs to stream both stdout and stderr to the console. Using Popen.communicate() seems to cause the outputs to block til the process is done if I redirect stdout and stderr to subprocess.PIPE. Using stdout=None and stderr=None with Popen.communicate() seems to stream to the console, so that�s good, but that leads me to�

2) I need to capture the contents of stdout and stderr in their own text files, preferably writing to them as the process continues.

Any idea how I do this? Sorry if it�s unclear I�m phone posting and also dumb

# ? Jun 7, 2021 17:20

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

punished milkman posted:

Hoping someone can help me with this because I�m out of my element and pretty stuck.

I need to make a subprocess calling function that has a few properties.

1) it needs to stream both stdout and stderr to the console. Using Popen.communicate() seems to cause the outputs to block til the process is done if I redirect stdout and stderr to subprocess.PIPE. Using stdout=None and stderr=None with Popen.communicate() seems to stream to the console, so that�s good, but that leads me to�

2) I need to capture the contents of stdout and stderr in their own text files, preferably writing to them as the process continues.

Any idea how I do this? Sorry if it�s unclear I�m phone posting and also dumb

Can you just not stream to stdout and instead stream to a file/logger and read the log file with something like notepad++ which will stream updates to files?

# ? Jun 7, 2021 17:45

HappyHippo: Nov 19, 2003; Do you have an Air Miles Card?

I think .communicate waits for the process to exit before returning? I think you can do something like

code:

proc = subprocess.Popen(...,stdin=subprocess.PIPE, stdout=subprocess.PIPE, bufsize=1)

and then read the lines out of proc one at a time. You can print the lines and also write them to a file.

# ? Jun 7, 2021 18:24

SurgicalOntologist: Jun 17, 2004

You can directly pass an open file to the stdin/stderr args of Popen any of the subprocess helper functions.

Edit: oops, misunderstood. I missed the console part. You want to do a tee basically. I don't know if there's any shortcut besides iterating. Well, maybe just use tee outside python.

SurgicalOntologist fucked around with this message at 19:23 on Jun 7, 2021

# ? Jun 7, 2021 19:15

post hole digger: Mar 21, 2011

Can someone help me with a stupid boto3 issue I'm dealing with? I have this code block using boto3's route 53 library to update a dns record in aws

code:

def create_route53_entry(host, ip):
    route53 = boto3.client('route53')
    try:
        response = route53.change_resource_record_sets(
            HostedZoneId='ZZZZZEXAMPLE',
            ChangeBatch={
                'Changes': [
                    {
                        'Action': 'CREATE':
                        'ResourceRecordSet': {
                            'Name': host,
                            'Type': 'A',
                            'TTL': 300,
                            'ResourceRecords': [{'Value': ip}]
                        }
                    }]
            })
    except Exception as e:
        print(e)


create_route53_entry('test.domain.com', '10.4.20.69')

when I try to run, I'm getting the error:

code:

    ChangeBatch={
    ^
SyntaxError: expression cannot contain assignment, perhaps you meant "=="?

I can't figure out why. It seems like I'm using the same format the examples in their docs recommend. anyone know what I'm doing wrong? is my whitespace somehow screwed up? It doesnt seem like thats it, but I'm really at a loss otherwise...

Edit: Very dumb typo... 'Action': 'CREATE': to 'Action': 'CREATE', fixed it.

post hole digger fucked around with this message at 20:42 on Jun 7, 2021

# ? Jun 7, 2021 20:12

necrotic: Aug 2, 2005; I owe my brother big time for this!

dict literal keys must be quoted in python

oh wait that's kwargs. I have no idea.

# ? Jun 7, 2021 20:42

SurgicalOntologist: Jun 17, 2004

Nevermind

# ? Jun 7, 2021 20:45

Data Graham: Dec 28, 2009; 📈📊🍪😋

Stupid parser tricks

When this happens it's funny to look at how it ended up with that unhelpful error message and try to figure out what it thought you were trying to pull off

# ? Jun 7, 2021 21:25

QuarkJets: Sep 8, 2008

punished milkman posted:

Hoping someone can help me with this because I�m out of my element and pretty stuck.

I need to make a subprocess calling function that has a few properties.

1) it needs to stream both stdout and stderr to the console. Using Popen.communicate() seems to cause the outputs to block til the process is done if I redirect stdout and stderr to subprocess.PIPE. Using stdout=None and stderr=None with Popen.communicate() seems to stream to the console, so that�s good, but that leads me to�

2) I need to capture the contents of stdout and stderr in their own text files, preferably writing to them as the process continues.

Any idea how I do this? Sorry if it�s unclear I�m phone posting and also dumb

Other posts have the right of it, I just wanted to add some more specific notes: this is made a little complicated by wanting to write the streams to separate files while printing both to the console. If you just wanted to do one or the other it'd be pretty easy.

I know how to get you most of the way there, at least: create a Popen with stderr piped to stdout (using subprocess.STDOUT as the argment for stderr, I believe) and then iterate over Popen.stdout. You can write lines wherever you want, iteration won't complete until the process exits, you can print to the terminal, the issue with this scheme is that stderr and stdout are combined.

I guess you could use PIPE for both stdout and stderr, then create a while loop that 1) does something with contents of stdout, 2) does something with contents of stderr, 3) repeat until both buffers are empty and poll() indicates that the process terminated. I've never tried doing something like this, and there may be a better way.

# ? Jun 8, 2021 00:14

i vomit kittens: Apr 25, 2019

I have a Flask/SQLAlchemy question that I'm having trouble figuring out the right combination of words to even Google to get help. I'm making a simple message board app for a class project and am trying to work out a query that will give me the following for a page of threads:

- The threads themselves.
- The number of posts in each thread.
- The created_at datetime of the most recent post in each thread.
- The username of the people who made those most recent posts.

I've managed to get the first three with this:

code:

    queries = [Thread.board_id == board_id]
    if request.args.get("search"):
        queries.append(Thread.title.ilike(f"%{request.args.get('search')}%"))
    threads = db.session \
        .query(
            Thread,
            func.max(Post.created_at).label("last_post_time"),
            func.count(Post.thread_id).label("post_count")
        ) \
        .join("posts") \
        .group_by(Thread) \
        .filter(*queries) \
        .order_by(desc(Thread.pinned), desc("last_post_time")) \
        .paginate(page=request.args.get("page", default=1, type=int), per_page=50)

What I believe I need to figure out is how to get the entire record for the post that's found by the func.max(Post.created_at).label("last_post_time") part. The posts table has a user_id foreign key that I would at least be able to use to query the users table, if not just join it directly somehow. If anyone has any ideas on how to do that I'd appreciate the help.

# ? Jun 8, 2021 02:05

punished milkman: Dec 5, 2018; would have won

QuarkJets posted:

Other posts have the right of it, I just wanted to add some more specific notes: this is made a little complicated by wanting to write the streams to separate files while printing both to the console. If you just wanted to do one or the other it'd be pretty easy.

I know how to get you most of the way there, at least: create a Popen with stderr piped to stdout (using subprocess.STDOUT as the argment for stderr, I believe) and then iterate over Popen.stdout. You can write lines wherever you want, iteration won't complete until the process exits, you can print to the terminal, the issue with this scheme is that stderr and stdout are combined.

I guess you could use PIPE for both stdout and stderr, then create a while loop that 1) does something with contents of stdout, 2) does something with contents of stderr, 3) repeat until both buffers are empty and poll() indicates that the process terminated. I've never tried doing something like this, and there may be a better way.

Yep this is more or less what I landed on. I toyed around with some asyncio and aiofile stuff for a bit and quickly got overwhelmed and moved back to some synchronous code with some success. As thanks to everyone, here�s a lovely photo of my code

# ? Jun 8, 2021 04:31

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

punished milkman posted:

a lovely photo of code

this is haram

# ? Jun 8, 2021 05:01

qsvui: Aug 23, 2003; some crazy thing

wouldn't it be easier to just copy-paste the code?

# ? Jun 9, 2021 02:56

Data Graham: Dec 28, 2009; 📈📊🍪😋

Tell that to the guys on my team who just relentlessly spam intricately marked-up screenshots faster than they can type apparently

# ? Jun 9, 2021 03:10

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Data Graham posted:

Tell that to the guys on my team who just relentlessly spam intricately marked-up screenshots faster than they can type apparently

I can�t imagine why anyone would ever need to screenshot code

Like�every communication platform supports code markdown blocks

# ? Jun 9, 2021 04:16

boofhead: Feb 18, 2021

CarForumPoster posted:

I can�t imagine why anyone would ever need to screenshot code

Like�every communication platform supports code markdown blocks

Screenshots are old news. I take a scrolling video with my phone using full filters so that nobody can steal my awesome code. You might laugh but do you know how many people copy my code? That's right, none

# ? Jun 9, 2021 07:43

HappyHippo: Nov 19, 2003; Do you have an Air Miles Card?

qsvui posted:

wouldn't it be easier to just copy-paste the code?

I guess not everyone wants to log into their somethingawful.com account on their work computer

# ? Jun 9, 2021 13:11

Data Graham: Dec 28, 2009; 📈📊🍪😋

CarForumPoster posted:

I can�t imagine why anyone would ever need to screenshot code

Like�every communication platform supports code markdown blocks

I've asked them to stop and they won't stop

I guess maybe it's because they can circle things in red and make big red arrows pointing to things and highlight stuff in yellow which is great for making me feel like an absolute dullard who needs to be communicated with via flashcards

# ? Jun 9, 2021 13:17

NinpoEspiritoSanto: Oct 22, 2013

HappyHippo posted:

I guess not everyone wants to log into their somethingawful.com account on their work computer

So use one of the several hundred pastebins or online repls

# ? Jun 9, 2021 13:56

wolrah: May 8, 2006; what?

Data Graham posted:

I've asked them to stop and they won't stop

I guess maybe it's because they can circle things in red and make big red arrows pointing to things and highlight stuff in yellow which is great for making me feel like an absolute dullard who needs to be communicated with via flashcards

Context is important here too, if they're doing this not to share the code with you but to point something out in something you already have it makes sense.

I send my colleagues marked up screenshots of config files or web interfaces all the time, but in the context of "this setting is here" or "holy poo poo look what I just found in $newcustomer's equipment that we'll need to unfuck". If I'm actually sharing code with them I use a code block as any reasonable person should.

# ? Jun 9, 2021 16:23

punished milkman: Dec 5, 2018; would have won

Bundy posted:

So use one of the several hundred pastebins or online repls

taking a terrible photo of your computer screen is much funnier

# ? Jun 9, 2021 16:24

QuarkJets: Sep 8, 2008

Bundy posted:

So use one of the several hundred pastebins or online repls

I would be hesitant to do that for any corporate codebase that's not open source

# ? Jun 9, 2021 17:59

OnceIWasAnOstrich: Jul 22, 2006

QuarkJets posted:

I would be hesitant to do that for any corporate codebase that's not open source

I wouldn't put anything on SomethingAwful that I wouldn't put on Pastebin. At least with Pastebin it isn't automatically indexed if you set it to unlisted.

# ? Jun 9, 2021 18:09

mr_package: Jun 13, 2000

Is there a way to use subprocess for internal functions? Or, a way to set timeout value to a function that's fairly simple (in the standard library) maybe with multiprocessing module? I'm running into a fun issue where DNS lookups for .local are ignoring socket.timeout value on Mac (probably due to bonjour integration / special handling), and the default timeout is 5 seconds. This is an automated test that hits failure case if I forgot to connect to VPN so I just want it to use a very short timeout instead of sitting there for 5 seconds and then failing.

As a workaround I've put a function that returns the value of socket.gethostbyname() into a separate file and the very first test is now a fixture that calls subprocess.check_output() with timeout=1 parameter on that file. Works perfectly, saving me four seconds. Amazing. Just wondering if there's a way to make subprocess/standard library do this with local functions as opposed to calling separate python shell command.

# ? Jun 9, 2021 21:42

OnceIWasAnOstrich: Jul 22, 2006

mr_package posted:

Is there a way to use subprocess for internal functions? Or, a way to set timeout value to a function that's fairly simple (in the standard library) maybe with multiprocessing module? I'm running into a fun issue where DNS lookups for .local are ignoring socket.timeout value on Mac (probably due to bonjour integration / special handling), and the default timeout is 5 seconds. This is an automated test that hits failure case if I forgot to connect to VPN so I just want it to use a very short timeout instead of sitting there for 5 seconds and then failing.

As a workaround I've put a function that returns the value of socket.gethostbyname() into a separate file and the very first test is now a fixture that calls subprocess.check_output() with timeout=1 parameter on that file. Works perfectly, saving me four seconds. Amazing. Just wondering if there's a way to make subprocess/standard library do this with local functions as opposed to calling separate python shell command.

You can create a multiprocessing.Process with the target= argument as the callable you want to run, ideally using a Pipe or Queue to feed your data back to the main process. You then call .start() and then .join(timeout) with whatever timeout you want before checking the Pipe/Queue and calling it a day. If your underlying callable releases GIL you can even do this with threading.Thread instead.

# ? Jun 9, 2021 22:56

QuarkJets: Sep 8, 2008

concurrent.futures is the sexy new(er) wrapper for multiprocessing and makes this sort of stuff pretty easy. The Future object that gets returned has a wait method with an optional timeout argument

# ? Jun 10, 2021 02:07

Foxfire_: Nov 8, 2010

OnceIWasAnOstrich posted:

You can create a multiprocessing.Process with the target= argument as the callable you want to run, ideally using a Pipe or Queue to feed your data back to the main process. You then call .start() and then .join(timeout) with whatever timeout you want before checking the Pipe/Queue and calling it a day. If your underlying callable releases GIL you can even do this with threading.Thread instead.

Be aware that by default, multiprocessing on Linux is broken and will randomly deadlock. You need to change the start method from fork to spawn. The MacOS and Windows defaults are already correct.

It's default behavior is to fork() to make a copy of the process, then keep using it without calling exec(). This has always been forbidden by POSIX, but is commonly done. It will generally work fine as long as absolutely everything in the process holds no locks at the moment the fork occurs, which you have no way of assuring that is true in general (you can't easily prove that some library doesn't make a thread that it uses internally)

# ? Jun 10, 2021 04:18

mr_package: Jun 13, 2000

I did some experiements with ThreadPoolExecutor and ProcessPoolExecutor. A few surprises, such as the behaviors of context managers, which wait for everything to finish before exiting. This is probably a good thing, I was just surprised that getting future.result(timeout=1) did not have any effect e.g. that raises TimeoutError, but does not __exit__ the context manager.

I was also surprised that you basically cannot easily kill threads/processes that you launch. There are third-party libraries and what appear-- to me-- to be very hack-y approaches to dealing with this to be found on SO. Perhaps they are fine, perhaps this is just the way it is done. This is my first look at concurrency, do other languages handle this pretty much the same way or is it Python-specific? I'm thinking for example if you were writing something in go or rust or C# or whatever and you wanted to terminate a process early, is it easier?

And if so, what become the best way to do this in Python? It seems like I find myself back at square one: how do you gracefully __exit__ from a method if it is taking too long? In the current code, you get error messages when you expect to have them, but the actual program does not exit until everything is finished. For a 5 seconds DNS timeout, it's liveable but what if there's something longer that you need to kill if it exceeds 60s, or 300s, or whatever?

code:

import concurrent.futures
import platform
import socket
import time

def get_ip() -> str:
    if not platform.system() == "Darwin":
        time.sleep(5)  # simulate 5s timeout if running on non-MacOS 
    return socket.gethostbyname("host.local")

def main() -> str:
    executor = concurrent.futures.ProcessPoolExecutor()
    try:
        return executor.submit(get_ip).result(timeout=1)
    except concurrent.futures._base.TimeoutError:
        raise RuntimeError("Timeout resolving hostname.")
    finally:
        print("Shutting down ProcessPool...")
        executor.shutdown(wait=False, cancel_futures=True)

if __name__ == "_main__":
    try:
        main()
    except RuntimeError as err:
        print(f"Unable to resolve host: {err}")

One gotcha: concurrent.futures._base.TimeoutError is not the same as TimeoutError (errno.ETIMEDOUT). I spent some time trying to figure this out and pretty sure I saw someone on SO with same problem ("Why isn't it catching TimeoutError"?)

# ? Jun 10, 2021 21:17

Adbot: ADBOT LOVES YOU

# ? May 26, 2024 09:53

QuarkJets: Sep 8, 2008

You can kill a running process by just calling cancel() on its Future. Tested code:

Python code:

import concurrent.futures
import platform
import socket
import time

def get_ip() -> str:
    if not platform.system() == "Darwin":
        time.sleep(5)  # simulate 5s timeout if running on non-MacOS 
    return socket.gethostbyname("localhost")

def main() -> str:
    with concurrent.futures.ProcessPoolExecutor() as executor:
        try:
            future = executor.submit(get_ip)
            time.sleep(1)
            future.cancel()
            return 0
            #return future.result(timeout=1)
        except concurrent.futures._base.TimeoutError:
            raise RuntimeError("Timeout resolving hostname.")
        finally:
            print("Shutting down ProcessPool...")
            executor.shutdown(wait=False, cancel_futures=True)

if __name__ == '__main__':
    print(main())

This exits after 1 second rather than 5

# ? Jun 11, 2021 03:07

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »