|
Chin Strap posted:Used all the time in Pandas too And standard library string methods
|
# ? Sep 26, 2023 01:16 |
|
|
# ? May 15, 2024 04:00 |
|
Falcon2001 posted:I'll be honest, Python imports are kind of a murky soup to me, despite working with it a bunch. I would also be interested if anyone had a video/blog post or something that explained it without getting into the deep minutiae of import logic, simply the 'if you run from X you pull in Y' level stuff. Python imports have some deep complexity but the hand-wavey basics are easy and will apply to like 99.9% of cases. Files ending in ".py" are modules. Directories containing a file named "__init__.py" are modules too. In a Python session the sys.path is simply a list of directories, and you can import any module found in one of these directories. Modules can be nested, so you can "import x.y.z" mimics directory structure x/y/z.py or x/y/z/, but only so long as each directory has an __init__.py in it. You can use "from" to extract part of an import into a shortened alias, or to extract only specific objects from a module. From can be used to extract a module but can also extract other objects (classes, functions, etc.) defined in that module. These are all equivalent: Python code:
Python code:
Python code:
You can use relative imports for importing things from one module into another, the syntax replicates the basics of a linux filesystem; "from ..buttz import fartz" checks for a module that is 2 parent directories up named "buttz" and tries to import an object named "fartz". Usually when people have import problems it's because they forgot to include a "__init__.py" somewhere, they accidentally created a circular import (e.g. two modules both try to import from each other), or they are trying to import from a module that isn't in sys.path. You should not be modifying sys.path yourself unless you have a good reason. "I just want this bit of code to work for the next few minutes" is a good reason. If this is code you'll be working on for awhile and sharing with other people what you probably really want to do is make a package that can be installed with pip, which takes like 2 minutes once you know how to do it.
|
# ? Sep 26, 2023 06:27 |
|
I feel like every definition I’ve had of imports paints them nicely. Then I ask one follow up question and it falls apart.
|
# ? Sep 26, 2023 07:44 |
|
A pydantic JSON serialization question. How do you deal with custom non-pydantic objects ? I want instances of the House class to be serialized to a simple fixed string, here's an example Python code:
It's used as a member on a bunch of objects which get serialized too, and I'd rather not define a model_config.json_encoder on each one (also I'm not sure it would work) I believe it can be accomplished by correctly writing House.__get_pydantic_core_schema__. I tried variants on this snippet too but couldn't get it right Python code:
rowkey bilbao fucked around with this message at 11:39 on Sep 26, 2023 |
# ? Sep 26, 2023 11:31 |
|
QuarkJets posted:Python imports have some deep complexity but the hand-wavey basics are easy and will apply to like 99.9% of cases. This seems pretty reasonable. I think that so many IDEs/etc handwave and handle the 'what directories are part of your path for the purposes of testing' that people just don't have to dig into this much - at work we use a custom sort of python build system that pulls in dependencies and it can be pretty complex, but also something you don't touch directly, just define and run.
|
# ? Sep 26, 2023 17:58 |
|
Why can't I just import datetime? Why do I always have to do 'from datetime import datetime, timedelta', etc?
|
# ? Sep 26, 2023 23:07 |
|
Big Dick Cheney posted:Why can't I just import datetime? Why do I always have to do 'from datetime import datetime, timedelta', etc? You can. You just have to then use "datetime.datetime" and "datetime.timedelta" instead of "datetime" and "timedelta".
|
# ? Sep 26, 2023 23:16 |
To be clear, datetime is just confusing because inside the module "datetime" there is another submodule called "datetime". Which sucks when you're digging through a project that sometimes imports the top-level one, sometimes the second-level one. Just because whoever wrote it had no regard for future them (aka them)
|
|
# ? Sep 26, 2023 23:31 |
|
from them import frustration
|
# ? Sep 27, 2023 00:22 |
|
I don't have time to import time from time
|
# ? Sep 27, 2023 00:25 |
|
Armitag3 posted:from them import frustration as usual
|
# ? Sep 27, 2023 00:26 |
|
I'm very upset that that is perfectly valid Python
|
# ? Sep 27, 2023 03:53 |
|
I especially hate package names that install a completely different module name. Pillow is the big offender for me, I hate that the module you import is actually named PIL. May as well call it PISS
|
# ? Sep 27, 2023 04:24 |
|
QuarkJets posted:I especially hate package names that install a completely different module name. Pillow is the big offender for me, I hate that the module you import is actually named PIL. May as well call it PISS At least Pillow has the excuse of it being a fork of PIL so the import is for backwards compatibility/historical reasons. Beautiful Soup 4 with its dumb "install beautifulsoup4 (but not beautifulsoup, because that's beautifulsoup 3!) and then import bs4" can go to hell, though.
|
# ? Sep 27, 2023 05:02 |
And things like that are mostly because of abandonware issues.
|
|
# ? Sep 27, 2023 10:49 |
|
Coming back to my previous question from another angle I have a simple class Python code:
Python code:
Python code:
My problem is, the Gem model can be used in several places, for example all those instances are valid Python code:
Python code:
Can I somehow "make" Gem into an Annotation ? This gets real murky for me. e: full example rowkey bilbao fucked around with this message at 14:10 on Sep 27, 2023 |
# ? Sep 27, 2023 13:58 |
|
Why don't you just base Gem on one of the Pydantic objects?
|
# ? Sep 27, 2023 14:07 |
|
It's got a bunch of extra logic that I'd rather not touch, also it's not really my call.
|
# ? Sep 27, 2023 14:13 |
|
You can do like Python code:
|
# ? Sep 27, 2023 14:16 |
|
Data Graham posted:And things like that are mostly because of abandonware issues. I would say "just replace the abandonware" but I think we all know it's load-bearing abandonware at this point.
|
# ? Sep 27, 2023 15:17 |
|
Chin Strap posted:You can do like Is this what you had in mind ? I have a similar issue. Python code:
rowkey bilbao fucked around with this message at 17:08 on Sep 27, 2023 |
# ? Sep 27, 2023 15:19 |
|
A) You've got to give error output if you want help B) This seems like a terrible class to work with. Can't you simplify things at all
|
# ? Sep 27, 2023 15:43 |
|
Macichne Leainig posted:I would say "just replace the abandonware" but I think we all know it's load-bearing abandonware at this point. It's one of the big downsides of open source ecosystems with a centralized package repository. When someone abandons "pillow" then the community can pick it up, but the package "pillow" still belongs to that abandoned project. So everybody has to work around that limitation.
|
# ? Sep 27, 2023 17:15 |
|
Chin Strap posted:A) You've got to give error output if you want help I edited the error message in my previous post. I'm poking around trying to find something that breaks the least amount of things. I liked your idea but the way I implemented it changes the instance type from Gem to typing._AnnotatedAlias, which breaks down at some point in the test suite. Is there a way to create a class that retains the initial characteristics while also containing the necessary information for pydantic to do its serialization like I want to ? What I haven't tried yet is to add a Gem__metadata__ member that replicates what I see when calling inspect.get_annotations on an annotated type.
|
# ? Sep 27, 2023 17:18 |
|
I've never tried it myself, but can you use Gem and BaseModel as mixins to make a new class?
|
# ? Sep 27, 2023 17:25 |
|
I suggest take a step back and review your whole type system, not just Gem. In particular soil. It doesn't make sense to me that it can be so many different things (string, dict of strings, dict of gems). I mean, I can imagine the reasons. But I think if you try to design a proper Soil type, rather than trying to capture the different possibilities of Soil with different arrangements of builtin types, you will end up simplifying things a lot. Edit: to be more specific, you don't really have a dict in the sense of mapping arbitrary keys to values. At least judging by your choice of "underground" in the example, that looks like a domain concept, not an arbitrary string. From another angle, imagine you have your data model set up, and your next task is to implement a method that checks whether a certain gem is anywhere in the asteroid. That's going to be a nightmare with all the combinations you'll have to check. SurgicalOntologist fucked around with this message at 21:20 on Sep 27, 2023 |
# ? Sep 27, 2023 21:10 |
|
Is there a simple way in Python to persist a data structure of some kind between runs, like a database? Like literally just saving a list of dictionaries to JSON and then saving it to a file would probably be fine and might be what I end up going with. I have a one-off project to kick off a ticket campaign, so I generally want to run it for a few, then check on them, and then continue, maybe tweaking my script between runs. Mostly I want to make sure that I capture any state like ticket IDs/etc as the script goes through and then can say 'if ticket_id: continue' to skip double-making tickets.
|
# ? Sep 28, 2023 07:28 |
|
Falcon2001 posted:Is there a simple way in Python to persist a data structure of some kind between runs, like a database? Like literally just saving a list of dictionaries to JSON and then saving it to a file would probably be fine and might be what I end up going with. do you want a database? sqlite is built in json is good too and honestly much better if you don't need an actual database for some reason, but i hear from backend developers all the time "QuarkJets my dick can't get hard unless I'm working with a database in all of my projects" so maybe that's something you require. If you're looking for key membership among rows of data then it sounds like this may be you
|
# ? Sep 28, 2023 07:45 |
|
QuarkJets posted:do you want a database? sqlite is built in I thank God every day I wake up and no longer am on call for databases loving up, so I think I'll just use a JSON file.
|
# ? Sep 28, 2023 08:47 |
|
Falcon2001 posted:I thank God every day I wake up and no longer am on call for databases loving up, so I think I'll just use a JSON file. Whatever device you're using to read this is almost certainly running multiple applications that use SQLite internally. That said if there'd be a benefit to this file being manually editable with a plain text editor other options may be better.
|
# ? Sep 28, 2023 14:15 |
|
I wouldn't think about this problem as how you save data to disk, but instead what form do you want this data to take as you're using it (probably just a list or set of ticket IDs?). You could pickle that and write that to disk, or output as JSON and write that. Don't over think it.
|
# ? Sep 28, 2023 14:31 |
|
To me it sounds like something you'd like to query. I've not used sqlite but if it is really that low of a bar to set up makes sense to me.
|
# ? Sep 28, 2023 14:34 |
|
This is going to have 80 or less "rows" so honestly a pickle or a json is probably perfectly fine. I'll take a look at sqlite though.
|
# ? Sep 28, 2023 17:00 |
|
On the subject of pickles, does anyone use shelf? It looks pretty nice.
|
# ? Sep 29, 2023 01:19 |
|
I wrote a script that ran beautifully for the first couple months, but in the past week or two it's gotten suddenly and significantly slower. I'm not sure if it's because of one of the updates I pushed but that's my current guess. I'm running it now with cProfile to see what it can show me, but does anyone have some good beginner tips for troubleshooting performance issues? It's kinda funny because I've been in support for years, and I have always hated and avoided troubleshooting performance issues, but now I can't avoid it
|
# ? Sep 30, 2023 21:06 |
|
Does it read through some data set? Has it grown over time?
|
# ? Sep 30, 2023 21:30 |
|
I personally recommend PyInstrument as a profiler because it only shows you the things where your code spends substantial time. Its HTML output ability is nice. IIRC cProfile shows you every operation/call, many of which will be irrelevant to the program's speed of execution. Anyway, your question is pretty broad. Sometimes the issue comes from a bit of code that works fine when your dataset is small and not so fine when it isn't. I once had a small section of code that was consuming 25% of the program's runtime, and it's because it was repeatedly calling min() on a very large, growing set. This is not only O(n), it was n operations every time. Tracking the minimum value through another means sped up that section over 500x, and that bit of code didn't even register as noteworthy in the profiler anymore.
|
# ? Sep 30, 2023 21:47 |
|
spiritual bypass posted:Does it read through some data set? Has it grown over time? It does read through sets of data, but the sets are always within a certain size range and the data the script operates on doesn't change much. And yeah sorry for how vague and broad it is, but I'm a relative beginner, especially when it comes to troubleshooting a performance issue, so I'm just looking for broad and basic starting points. I'll give pyinstrument a look, thanks! Son of Thunderbeast fucked around with this message at 02:37 on Oct 1, 2023 |
# ? Oct 1, 2023 01:51 |
|
If it's gotten significantly slower recently, the first thing I'd do is grab an older version of the script that predates the slowdown. This does two things: - Lets you confirm that the slowdown was caused by a code change (or tells you to look elsewhere if it turns out the old version has also mysteriously gotten slower!) - Gives you a baseline for comparing profiles - if something takes a long but roughly equal time in both profiles, then it might be a good candidate for general optimization, but isn't actually the cause of the regression you're investigating. If something takes a long time with the new code but not with the old one then that's where you want to focus. Jabor fucked around with this message at 02:24 on Oct 1, 2023 |
# ? Oct 1, 2023 02:22 |
|
|
# ? May 15, 2024 04:00 |
|
Jabor posted:If it's gotten significantly slower recently, the first thing I'd do is grab an older version of the script that predates the slowdown. Between this and initial profiling tests, I think I found the issue--the slowdowns started around the time I added an extremely hacky dupe check to my already extremely hacky index ID generator, which gets called a lot. I knew when I was writing this that it was hosed up, because I was tired and googling libraries and shoving parts in and twisting them until they fit and did the thing I wanted. It worked fine enough except about once every 250,000 calls or so, it would throw an exception because of a hash collision. So I threw the while loop in there as a quick duct tape job. I'm going to work on fixing this, but if anyone has some tips I'm all ears. code:
Son of Thunderbeast fucked around with this message at 03:32 on Oct 1, 2023 |
# ? Oct 1, 2023 03:26 |