Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

Bruegels Fuckbooks: Sep 14, 2004; Now, listen - I know the two of you are very different from each other in a lot of ways, but you have to understand that as far as Grandpa's concerned, you're both pieces of shit! Yeah. I can prove it mathematically.

Proteus Jones posted:

Also lazy as gently caress.

you begin to appreciate lazy software people when you have to deal with software people who are not lazy, because the non-lazy software people generally try to solve their issues with lots and lots and lots of loving code and loving documentation to go along with that code, and their poo poo never works in the end anyway.

# ? Jun 20, 2020 16:27

Adbot: ADBOT LOVES YOU

# ? May 17, 2024 07:25

Spime Wrangler: Feb 23, 2003; Because we can.

Being lazy is extremely important. Much of being a good developer is about making it as easy as possible to be as lazy as possible in the future. Bonus points if you help others be lazy in the future too.

# ? Jun 20, 2020 17:34

Dr Subterfuge: Aug 31, 2005; TIME TO ROC N' ROLL

The reason I know I'm only okay at best at programming is it's way too hard to add things to the stuff I've already written.

# ? Jun 20, 2020 17:38

qsvui: Aug 23, 2003; some crazy thing

Spime Wrangler posted:

Being lazy is extremely important. Much of being a good developer is about making it as easy as possible to be as lazy as possible in the future. Bonus points if you help others be lazy in the future too.

"be long term lazy"

# ? Jun 20, 2020 18:41

Macichne Leainig: Jul 26, 2012; by VG

QuarkJets posted:

Yeah the entirety of the software development world is pretty lovely

I've had some great coworkers and at neither of the jobs I've worked would anyone allow GIMP to be included as a dependency of any project, and would have been called out as part of the code review process.

You seem like you're a blast to work with.

# ? Jun 20, 2020 19:18

QuarkJets: Sep 8, 2008

Protocol7 posted:

I've had some great coworkers and at neither of the jobs I've worked would anyone allow GIMP to be included as a dependency of any project, and would have been called out as part of the code review process.

You seem like you're a blast to work with.

lol yeah surely I'm just frustrating to work with because I tell people to not have Gimp as a dependency in production

# ? Jun 20, 2020 22:27

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Protocol7 posted:

I've had some great coworkers and at neither of the jobs I've worked would anyone allow GIMP to be included as a dependency of any project, and would have been called out as part of the code review process.

You seem like you're a blast to work with.

QuarkJets posted:

lol yeah surely I'm just frustrating to work with because I tell people to not have Gimp as a dependency in production

You all seem to agree with each other and both disagree with a straw man (who is based on my post, though I, too, agree with you)

# ? Jun 20, 2020 23:14

Macichne Leainig: Jul 26, 2012; by VG

QuarkJets posted:

lol yeah surely I'm just frustrating to work with because I tell people to not have Gimp as a dependency in production

It might be that you�re trying too hard to be funny with the sarcastic posts that you didn�t notice I�ve been agreeing with you this whole time that GIMP doesn�t belong in production?

# ? Jun 21, 2020 14:05

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

A team at a company I used to work for ran a web service that involved hosting user-provided 3d models. For a while at least, to generate the thumbnails (that's it - just the thumbnails) they would send a newly-uploaded model over to some aws loadbalancer that would spin up an instance and invoke blender on it with a predefined scene to render three sizes of thumbnails. When you uploaded a new model the thumbnails could take actual hours before they showed up (depending on the queue length) with no indication anything was happening.

Not python related but literally what we're talking about here lol

# ? Jun 21, 2020 14:11

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Phobeste posted:

A team at a company I used to work for ran a web service that involved hosting user-provided 3d models. For a while at least, to generate the thumbnails (that's it - just the thumbnails) they would send a newly-uploaded model over to some aws loadbalancer that would spin up an instance and invoke blender on it with a predefined scene to render three sizes of thumbnails. When you uploaded a new model the thumbnails could take actual hours before they showed up (depending on the queue length) with no indication anything was happening.

Not python related but literally what we're talking about here lol

And what they should�ve done is use an AWS Lambda function! Only issue.

# ? Jun 21, 2020 14:34

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

CarForumPoster posted:

And what they should�ve done is use an AWS Lambda function! Only issue.

A guy comes to the county courthouse looking to change his name. �Oh, what�s your current name?� asks the clerk, and when the man replies, �Jim Hitler� the clerk understands completely. �So what do you want to change it to?� asks the clerk. �Well, I�m thinking about Joe.�

# ? Jun 21, 2020 14:42

QuarkJets: Sep 8, 2008

Protocol7 posted:

It might be that you�re trying too hard to be funny with the sarcastic posts that you didn�t notice I�ve been agreeing with you this whole time that GIMP doesn�t belong in production?

I understood that already. In fact I'm not disagreeing with you at all! I'm just pointing out the widespread notion of software development being a shitshow. Like even NASA source code from the Apollo program was full of "OH GOD WTF I HOPE THIS WORKS" comments, it's no secret that badly-designed proof of concept systems wind up in production all the time. The very forums we're posting on now are proof of that

I don't know or care why pointing this out led to you making a milquetoast personal attack but I hope this post helps you to understand what I was saying a little better.

# ? Jun 21, 2020 15:03

Macichne Leainig: Jul 26, 2012; by VG

So be the change you want to see in the world and don�t let your coworkers check in lovely proof of concept code into production. Which has also been my point.

# ? Jun 21, 2020 15:19

Dominoes: Sep 20, 2007

There are good parts of NASA programming too:

Richard Feynman posted:

The software is checked very carefully in a bottom-up fashion. First, each new line of code is checked, then sections of codes or modules with special function are verified. The scope is increased step by step until the new changes are incorporated into a complete system and checked. This complete output is considered the final product, newly released. But completely independently there is an independent verification group, that takes an adversary attitude to the software development group, and tests and verifies the software as if it were a customer of a delivered product. There is additional verification in using the new programs in simulators, etc. A discovery of an error during the verification testing is considered very serious, and its origin studied very carefully to avoid such mistakes in the future. Such unexpected errors have been found only about six times in all the programming and program changing (for new or altered payloads) that have been done. The principle that is followed is that all the verification is not an aspect of program safety, it is merely a test of that safety, in a non-catastrophic verification. Flight safety is to be judged solely on how well the programs do in the verification tests. A failure here generates considerable concern.

To summarize, then, the computer software checking system and attitude is of highest quality. There appears to be no process of gradually fooling oneself while degrading standards so characteristic of the Solid Rocket Booster or Space Shuttle Main Engine safety systems. To be sure, there have been recent suggestions by management to curtail such elaborate and expensive tests as being unnecessary at this late date in Shuttle history. This must be resisted for it does not appreciate the mutual subtle influences, and sources of error generated by even small changes of one part of a program on another. There are perpetual requests for changes as new payloads and new demands and modifications are suggested by the users. Changes are expensive because they require extensive testing. The proper way to save money is to curtail the number of requested changes, not the quality of testing for each.

This praise is in stark contrast to his assessment of other NASA teams.

# ? Jun 21, 2020 15:28

Hed: Mar 31, 2004; Fun Shoe

Random question--is there a way to turn off PyCharm's duplicated code fragment warning just in tests? When I'm in a test_*.py file, I'd want different rules.

There's usually a bit of boilerplate in my testing code that I want to explicitly repeat. I know I could suppress but no guarantees everyone is using PyCharm with my settings.

# ? Jun 21, 2020 21:15

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Hed posted:

Random question--is there a way to turn off PyCharm's duplicated code fragment warning just in tests? When I'm in a test_*.py file, I'd want different rules.

There's usually a bit of boilerplate in my testing code that I want to explicitly repeat. I know I could suppress but no guarantees everyone is using PyCharm with my settings.

I use maybe 10% of the automation PyCharm can do so IDK but could you fix the repeated boilerplate problem with a function or a function + partial?

# ? Jun 21, 2020 22:05

Macichne Leainig: Jul 26, 2012; by VG

According to the JetBrains site you can disable the inspection for certain files.

https://www.jetbrains.com/help/idea/analyzing-duplicates.html

The specific section is �Show duplicates from,� but I don�t know if that has the same problem with suppressions that you mentioned.

Seems like it�d be nice to have the ability to insert a comment to disable the inspection on a file but I don�t see that functionality unfortunately.

# ? Jun 22, 2020 00:19

KICK BAMA KICK: Mar 2, 2009

Protocol7 posted:

Seems like it�d be nice to have the ability to insert a comment to disable the inspection on a file but I don�t see that functionality unfortunately.

code:

# noinspection DuplicatedCode

is what gets inserted at the function or class level when you select the option to suppress from the inspection results but idk if it works at the top of a file.

# ? Jun 22, 2020 00:31

Wallet: Jun 19, 2006

CarForumPoster posted:

I use maybe 10% of the automation PyCharm can do so IDK but could you fix the repeated boilerplate problem with a function or a function + partial?

The duplicated code warning has literally never alerted me to duplicated code that I wasn't very much aware of, but if you're doing the same boilerplate at the top of every test that seems like a good use case for a fixture (or the equivalent for whatever you're using to test).

# ? Jun 22, 2020 01:59

Macichne Leainig: Jul 26, 2012; by VG

KICK BAMA KICK posted:

code:
# noinspection DuplicatedCode
is what gets inserted at the function or class level when you select the option to suppress from the inspection results but idk if it works at the top of a file.

I just checked in a file real quick and it does not do anything when placed at the top unfortunately. You're only given the option to suppress at the class, function or statement level.

In the settings menu you can change whether it detects duplicates only within a single file, or across the whole project, and also set a threshold for skipping duplicates that are smaller than a certain size (like copy-pasted 1-liners, I guess).

You can save a project-level inspection profile, though, it will save a profile XML file in the .idea folder, but I don't know if that will work for your purposes Hed, and still doesn't solve the inspection skipping specific files problem.

# ? Jun 22, 2020 04:28

Hed: Mar 31, 2004; Fun Shoe

Thanks all, I'm going to annotate with the noinspect on the few test classes where I'm repeating myself. I like the fixture idea but this hasn't swollen to enough tests now to warrant that, and I've gotten in trouble making my tests too complex in the past

# ? Jun 22, 2020 14:22

Cyril Sneer: Aug 8, 2004; Life would be simple in the forest except for Cyril Sneer. And his life would be simple except for The Raccoons.

Trying to get a question in here before the forums implode :v:

I'm working on a Windows application where I have a GUI that plays a movie. I capture every (say) 5th frame, then pass this image to a machine learning module that identifies if it contains a cat or not. If it does contain a cat, it goes on for further image manipulation (dresses the cat up in a a goofy costume for example). The ML identification task takes a "long" time, and the image manipulation task takes a "long" time - long in the sense that it takes much longer than my frame update rate - which is fine, I don't need my dressed-up cat images in real time. However, I do not want these processing steps blocking my movie playback.

So I've been thinking I need some type of FIFO queue approach, where each new image gets pushed as a task into a queue. But that's about where my knowledge runs out. Searching for queues brings up all kinds of things, plain old queues, queues with multiple threads, queues this multiple process, just multiprocessing, just multithreading, asyncio...blarrgh it all kind of goes over my head.

As a further complication, I'm using PyQt5 and pyqtgraph, so, whatever my design pattern is, it has to be able to drive/connect/trigger GUI updates safely (so it probably has to play nicely with signals and slots).

Thanks goons!

# ? Jun 25, 2020 18:33

KICK BAMA KICK: Mar 2, 2009

So yeah speaking of is there a Discord/exit strategy for CoC? Learned a ton here, would hate to lose it.

Cyril Sneer posted:

Is a task queue what you're looking for -- main program enqueues your long-running functions, which are executed in a separate worker process? Sounds similar to a question I asked a while ago and that was the answer. python-rq is probably the simplest one to try first (you'll need a Redis server, but that's trivial with Docker and a great excuse to learn if it you haven't). Don't think it pings back on job completion out of the box so your main loop would probably query the queue periodically to see what's finished and then act accordingly.

# ? Jun 25, 2020 21:53

Cyril Sneer: Aug 8, 2004; Life would be simple in the forest except for Cyril Sneer. And his life would be simple except for The Raccoons.

KICK BAMA KICK posted:

So yeah speaking of is there a Discord/exit strategy for CoC? Learned a ton here, would hate to lose it.

Is a task queue what you're looking for -- main program enqueues your long-running functions, which are executed in a separate worker process? Sounds similar to a question I asked a while ago and that was the answer. python-rq is probably the simplest one to try first (you'll need a Redis server, but that's trivial with Docker and a great excuse to learn if it you haven't). Don't think it pings back on job completion out of the box so your main loop would probably query the queue periodically to see what's finished and then act accordingly.

Thanks, I think this might be a bit overkill. I actually went ahead and just used python's own threading and queue libraries ( https://docs.python.org/3/library/queue.html ), and this seems to work, though I have yet to wire it up to my GUI.

# ? Jun 25, 2020 22:49

QuarkJets: Sep 8, 2008

Cyril Sneer posted:

Thanks, I think this might be a bit overkill. I actually went ahead and just used python's own threading and queue libraries ( https://docs.python.org/3/library/queue.html ), and this seems to work, though I have yet to wire it up to my GUI.

Nice! I was going to suggest this route, just a single queue and a single thread running the combination of "analyze image" and "process image". Qt is designed to operate in this manner anyway, wherein the main thread runs the GUI and spawned threads perform heavier processing loads. With GPUs commonly being used for machine learning (and hopefully you use one for image processing) it's almost like you're escaping from the performance restrictions imposed by the GIL, it's like a performance vs effort sweet spot.

You have the option of using a QThread or a standard python thread. QThreads are QObjects, and will emit signals when a thread is started or finished, which is a cool feature that is sometimes useful; it sounds like you don't have any use for that in this case, but it's good to be aware of these features. Your main thread should have no issue receiving signals emitted by your worker thread regardless of which kind it is, so long as it's all in the same process

# ? Jun 26, 2020 00:04

Cyril Sneer: Aug 8, 2004; Life would be simple in the forest except for Cyril Sneer. And his life would be simple except for The Raccoons.

QuarkJets posted:

Nice! I was going to suggest this route, just a single queue and a single thread running the combination of "analyze image" and "process image". Qt is designed to operate in this manner anyway, wherein the main thread runs the GUI and spawned threads perform heavier processing loads. With GPUs commonly being used for machine learning (and hopefully you use one for image processing) it's almost like you're escaping from the performance restrictions imposed by the GIL, it's like a performance vs effort sweet spot.

The training was done on a GPU but the model itself is small, so predictions run fine on standard CPU hardware. As does the image processing, though not quite in real time.

QuarkJets posted:

You have the option of using a QThread or a standard python thread. QThreads are QObjects, and will emit signals when a thread is started or finished, which is a cool feature that is sometimes useful; it sounds like you don't have any use for that in this case, but it's good to be aware of these features. Your main thread should have no issue receiving signals emitted by your worker thread regardless of which kind it is, so long as it's all in the same process

When a given image processing task finishes, I do want to trigger some GUI updates and as you know, you're not supposed to call a GUI update directly. So I will have to find a way to emit Q-compatible signals. Presumably if I use QThreads from the get-go, this should be relatively straightforward.

# ? Jun 26, 2020 02:12

QuarkJets: Sep 8, 2008

I recommend just creating your own signal and then emit it whenever you need to perform a gui update, connecting it before your thread starts. With the intrinsic QThread signals you'd have to restart the thread each time a signal is emitted, which is fine but might be more work than just writing your own signal

# ? Jun 26, 2020 03:37

Fluue: Jan 2, 2008

Question about abstracting over dependencies (API clients in this case) to keep imports lightweight and free of side-effects.

I'm trying to write a module that lets me make calls like employee.find(email="a@a.com") which is an abstraction around a directory API containing employees.This API requires credentials which I store in AWS SSM Parameter Store because this is part of a Lambda application.

The problem I'm running into is setting this up so that importing employee.py does not trigger a call to AWS. I would also like to maintain an instance of the client so that connections are reused so that calling employee.find() elsewhere in the code doesn't create a new connection or re-init the API class.

I was thinking that creating a singleton instance in the my_api.py module would bet the way to go, but passing credentials/options would still require a call to Parameter Store during the first import.

Alternatively, I have an example of lazy-loading auth/config below, but it feels like testing this would be difficult:

code:

# employee.py
from .my_api import MyDirectoryAPI
from .models import Employee

directory = MyDirectoryAPI()


def find(email):
    data = directory.get_by_email()
    return Employee.load(**data)

#########################################

# my_api.py
from . import config

class MyDirectoryAPI:

    def __init__(self):
        self._auth = None
    
    def _request(self, *args, **kwargs):
        auth = self._get_auth()
        return do_request_ex(url, method, auth=auth)

    def find_by_email(self, email):
        # do stuff

    def _get_auth(self):
        if self._auth is not None:
            return self._auth

        # makes remote call to AWS SSM via boto3
        conf = config.from_ssm(key="myapi")
        auth = {"user": conf["username"], "token": conf["private_key"]}
        
	# ...etc...
        # sometimes an api would require building a credentials object instead

        self._auth = auth
        return auth

Am I approaching client re-use and side-effect imports the wrong way here? A lot of lambda applications define the single instance at module level in the handler, but this client and abstraction are called in other modules.

# ? Jul 1, 2020 06:33

KICK BAMA KICK: Mar 2, 2009

Humble Bundle with PyCharm Professional licenses, if you've ever been curious about checking it out.

# ? Jul 1, 2020 19:29

hhhmmm: Jan 1, 2006; ...?

KICK BAMA KICK posted:

Humble Bundle with PyCharm Professional licenses, if you've ever been curious about checking it out.

Pycharm is a good editor.

# ? Jul 1, 2020 19:56

punished milkman: Dec 5, 2018; would have won

hhhmmm posted:

Pycharm is a good editor.

PyCharm pro is amazing and I use it every day at work, JetBrains makes wicked software. We actually just recently started using YouTrack which also owns (as much as a stupid agile issue tracker software can own I guess). Haven't been able to convince anyone to buy me a DataGrip license yet, but I'm working on it.

# ? Jul 1, 2020 20:09

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

hhhmmm posted:

Pycharm is a good editor.

Definitely.

Being a teacher I get free access to all the JetBrains stuff. It's quite lovely, although to be honest I'm pretty sure I don't use any of the features of the pro versions.

# ? Jul 1, 2020 22:07

Dominoes: Sep 20, 2007

I use it for Python and beyond. By far the best editor I've used

# ? Jul 1, 2020 22:57

Cosa Nostra Aetate: Jan 1, 2019

punished milkman posted:

PyCharm pro is amazing and I use it every day at work, JetBrains makes wicked software. We actually just recently started using YouTrack which also owns (as much as a stupid agile issue tracker software can own I guess). Haven't been able to convince anyone to buy me a DataGrip license yet, but I'm working on it.

Doesn't pycharm pro include datagrip? I was surprised at how much I liked it once I got pro through work--i used SQL all the time in my "analytics" world, but being able to mock the DB easily and validate things was super nice.

# ? Jul 2, 2020 00:45

susan b buffering: Nov 14, 2016

2013 lurker rereg posted:

Doesn't pycharm pro include datagrip? I was surprised at how much I liked it once I got pro through work--i used SQL all the time in my "analytics" world, but being able to mock the DB easily and validate things was super nice.

All of Jetbrains IDEs contain the features of DataGrip, yeah.

# ? Jul 2, 2020 00:58

Wallet: Jun 19, 2006

punished milkman posted:

PyCharm pro is amazing and I use it every day at work, JetBrains makes wicked software.

Yeah, PyCharm is excellent, even for non python stuff. It's wild how broad and consistently executed its feature set is.

# ? Jul 2, 2020 01:02

Proteus Jones: Feb 28, 2013

I pay for it myself. I'm not usually a fan of sub models that are more than $20/yr or so, but PyCharm is one of the exceptions for me. Worth every single penny.

# ? Jul 2, 2020 01:26

death cob for cutie: Dec 30, 2006; dwarves won't delve no more
too much splatting down on Zot:4

Wait, DataGrip is part of PyCharm? I've been paying for JetBrains Fuckin' Everything for like 20 bucks a month to get both PyCharm and DataGrip

# ? Jul 2, 2020 01:39

punished milkman: Dec 5, 2018; would have won

2013 lurker rereg posted:

Doesn't pycharm pro include datagrip? I was surprised at how much I liked it once I got pro through work--i used SQL all the time in my "analytics" world, but being able to mock the DB easily and validate things was super nice.

What really? Guess Pycharm is even more jacked up than I thought. Definitely going to test out the DB integration tomorrow

# ? Jul 2, 2020 01:42

Adbot: ADBOT LOVES YOU

# ? May 17, 2024 07:25

KICK BAMA KICK: Mar 2, 2009

Wallet posted:

Yeah, PyCharm is excellent, even for non python stuff. It's wild how broad and consistently executed its feature set is.

Yeah I keep my stuff in two separate projects like MyAppCore with all my Django code, which PyCharm Professional is great with, and then another MyAppStack with my Docker Compose dot yamls and all the Dockerfiles, bash scripts and .envs for the various services like postgres and nginx and PyCharm also handles all those excellently (with the right plugins).

# ? Jul 2, 2020 03:36

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »