Question on how best to handle shared data between two python modules. I have a class that holds state information within a dict and I have 2 modules using this class to read and modify the dict data but right now they each have their own separate objects of this class so the state data within the dict isn't shared. I'd like there to be one shared object of this class that each module can access. What is the Pythonic way of doing this?
|
|
# ? Feb 8, 2022 20:11 |
|
|
# ? May 15, 2024 16:20 |
|
I just discovered "pickling" Python objects - and that SQLAlchemy supports a Pickle field type natively. Is it worth trying to pickle objects or is it better just to serialize them as JSON and **dict them back to objects when loading them?
|
# ? Feb 8, 2022 20:20 |
|
Epsilon Plus posted:How should I handle a situation in which code wants to create an instance of a class, but shouldn't because it's being passed the wrong data? There are a lot of ways to tackle this, but probably the easiest would be Python code:
If you have the time, it would be better to use something like https://github.com/python-desert/desert to generate serialization schemas and validate against that schema while constructing. I assume you're doing some kind of network stuff here as well, but if you're not, you can also use a static type checker to make sure that your codebase isn't passing anything that isn't an integer to this function. Popete posted:Question on how best to handle shared data between two python modules. I have a class that holds state information within a dict and I have 2 modules using this class to read and modify the dict data but right now they each have their own separate objects of this class so the state data within the dict isn't shared. I'd like there to be one shared object of this class that each module can access. What is the Pythonic way of doing this?
|
# ? Feb 8, 2022 20:21 |
|
12 rats tied together posted:There are a lot of ways to tackle this, but probably the easiest would be Isn't the instance of the object still created at that point, though? Wouldn't I just end up with an uninitialized/partially initialized object of that class? I mean if it raises an exception right then and brings down the whole program it's not really an issue anymore per se, but I don't want the entire program to fall over irrecoverably over it.
|
# ? Feb 8, 2022 20:28 |
|
Crossposting from the SQL questions thread. I'm working on a Flask app to improve workflows for me and as coding/webdev practice, using SQLite3/SQLAlchemy as the backend. I have a table color with the fields id, name, abbrev. This is intended to be a lookup table and something that I could pull off easily in Access. I also have a table storm_panel_header with fields id, customer, job_no, header_color, sill_color, line_items where line_items is the one side of a one-to-many relationship with StormPanelLines. Where I am running into trouble is defining the type of relationship between color and storm_panel_header. Both header_color and sill_color need to get their values from color.id, but color is also intended, like I said, as a lookup table. That is to say, storm_panel_header will not be the one and only table referring to color by color.id. I cannot figure out how to assign the foreign key and backref between the two and it's pissing me off. EDIT: If I specify a ForeignKey relationship on storm_panel_header.header_color and storm_panel_header.sill_color but no backref in color, does that essentially make it a one-way lookup? I don't ever foresee a situation in which I am going to need to look up what items across several tables have which color. D34THROW fucked around with this message at 21:53 on Feb 8, 2022 |
# ? Feb 8, 2022 21:47 |
I’m not sure I understand the problem. You don’t know how to generate foreign keys in code, or how to model your problem in SQL? The latter is trivial, header_color_id and still_color_id are foreign keys referencing color.id. The real problem your schema is line_items. Storing an array there can cause giant problems for certain query patterns, and the way to prevent that is to have separate table called line_item with fields id, storm_panel_header_id, and storm_line_item_id, where the latter two are foreign keys of id fields of respective tables. cinci zoo sniper fucked around with this message at 22:01 on Feb 8, 2022 |
|
# ? Feb 8, 2022 21:57 |
|
Epsilon Plus posted:Isn't the instance of the object still created at that point, though? Wouldn't I just end up with an uninitialized/partially initialized object of that class? I mean if it raises an exception right then and brings down the whole program it's not really an issue anymore per se, but I don't want the entire program to fall over irrecoverably over it. Python object initialization isn't done until __init__ is complete, so this Python code:
In 99.99% of cases you shouldn't use __new__ and if you're in the .01% you'll know it, guido. If you're worried about saving memory, hey it's a gc'd language it'll go away eventually probably
|
# ? Feb 8, 2022 21:59 |
|
cinci zoo sniper posted:I’m not sure I understand the problem. You don’t know how to generate foreign keys in code, or how to model your problem in SQL? Oh, storm_panel_lines is going to have fields for enough data to compute the particulars for a report, id, header_id, width, height, trapped, top_attach, bottom_attach. That's enough for the actual class object to do the lookups and math. I think I was overcomplicating things. Any individual color doesn't need to know what line items have its id so I don't think I need to backref() that table to its "children". EDIT: line_items holds the relationship of the header (parent) to its children in StormPanelLines - the one side of the one-to-many of header-to-lines. D34THROW fucked around with this message at 22:15 on Feb 8, 2022 |
# ? Feb 8, 2022 22:12 |
If lines has header reference, I have no idea why you want to reference lines in header at all.
|
|
# ? Feb 8, 2022 22:26 |
|
D34THROW posted:I just discovered "pickling" Python objects - and that SQLAlchemy supports a Pickle field type natively. Is it worth trying to pickle objects or is it better just to serialize them as JSON and **dict them back to objects when loading them? If you have a dict of basic types then that already maps so well to json that you should just write a json, but if you have some complicated object that you need to store and load later then you could pickle it. I don't like writing entire objects to disk and would rather convert some of the information going in/out, but I'm sure there are situations I haven't imagined where I would be like "o yeah pickling this makes sense"
|
# ? Feb 9, 2022 00:10 |
|
Popete posted:Question on how best to handle shared data between two python modules. I have a class that holds state information within a dict and I have 2 modules using this class to read and modify the dict data but right now they each have their own separate objects of this class so the state data within the dict isn't shared. I'd like there to be one shared object of this class that each module can access. What is the Pythonic way of doing this? What you're looking for is a class attribute, but what you're describing (a module secretly modifying shared data being used by another module, but without being called directly?) is frustrating to debug and has poor readability. You should define functions that mutate data in a clear and concise way. Have one of these modules pass an instance of the class to the other module for mutation
|
# ? Feb 9, 2022 00:21 |
|
Epsilon Plus posted:Isn't the instance of the object still created at that point, though? Wouldn't I just end up with an uninitialized/partially initialized object of that class? I mean if it raises an exception right then and brings down the whole program it's not really an issue anymore per se, but I don't want the entire program to fall over irrecoverably over it. Raising an exception doesn't mean that the program falls over irrecoverably, the exception just has to be caught. Raising an exception for bad inputs is the most commonly used and also best way to do what you want to do
|
# ? Feb 9, 2022 00:27 |
|
Phobeste posted:Python object initialization isn't done until __init__ is complete, so this poo poo, I thought it was done when __new__ was complete... which would rely on __init__ being complete... ... why did I need to ask this question, again? I could have sworn it creates an instance, then uses __init__ to populate it afterwards.
|
# ? Feb 9, 2022 01:51 |
|
Epsilon Plus posted:poo poo, I thought it was done when __new__ was complete... which would rely on __init__ being complete... i'm pretty sure that's how it works internally but when you're creating an object they both get done at the same sequence point relative to the creator invocation so if either fails the result won't get bound
|
# ? Feb 9, 2022 02:20 |
|
Ok, I've been trying to figure this out for a bit and I'm stumped. I'm writing a test for the simple timer function below. I'd like to patch time.sleep() so that countdown_timer is called in the test it basically runs as a nothing function and takes essentially no time at all so that I can test larger numbers to ensure the counter is working without actually waiting the timer to sleep. I'm looking through explanations of how to do this, but I'm a bit stumped?Python code:
BeastOfExmoor fucked around with this message at 08:49 on Feb 9, 2022 |
# ? Feb 9, 2022 06:07 |
|
Something like this should work:Python code:
|
# ? Feb 9, 2022 07:03 |
|
Thread, I interview Python coders. I often get new grads who list themselves as Python (Advanced) in their skills section. 100% of them I am disappointed with what I feel like are basic questions about how Python is used here in practice, which means it might be me. (For example, describe how you might use and anonymous or lambda function.) I've basically stopped asking ANY coding questions in the initial interview, instead favoring reviewing their GitHub before a phone interview then a short web app project after the phone interview. (Call up this no-auth-required API, display results in a sortable table. Here are a list of requirements.) Google results for "python interview questions" have quiz-like questions ABOUT the language that I feel aren't great for seeing who can learn quickly and has spent 1000s of hours to become an advanced level practitioner. Are there any good questions you'd ask someone who describes themselves as a Python expert that wouldn't cut out people who simply used Python in a different way? I've so far been very happy with the quality of my hires, but always looking to get better. CarForumPoster fucked around with this message at 16:01 on Feb 9, 2022 |
# ? Feb 9, 2022 15:55 |
|
Edit: I need to refresh
|
# ? Feb 9, 2022 15:58 |
CarForumPoster posted:Thread, I interview Python coders. I often get new grads who list themselves as Python (Advanced) in their skills section. 100% of them I am disappointed with what I feel like are basic questions about how Python is used here in practice, which means it might be me. (For example, describe how you might use and anonymous or lambda function.) I've basically stopped asking ANY coding questions in the initial interview, instead favoring reviewing their GitHub before a phone interview then a short web app project after the phone interview. (Call up this no-auth-required API, display results in a sortable table. Here are a list of requirements.) I personally don’t believe that asking language trivia on an interview is all that useful, but if you must insist, this is a better filter than just mauling the applicant flow with a barbed LeetCode, I guess. My 5 minute hot take on possible questions for someone aiming for a generic senior Python role: 1) When are context managers useful? 2) What are decorators in Python? When would you use one? 3) What are advantages and disadvantages of pickling? 4) What is GIL? What can you do in a situation where it is a performance bottleneck?
|
|
# ? Feb 9, 2022 16:27 |
|
CarForumPoster posted:Thread, I interview Python coders. I often get new grads who list themselves as Python (Advanced) in their skills section. 100% of them I am disappointed with what I feel like are basic questions about how Python is used here in practice, which means it might be me. (For example, describe how you might use and anonymous or lambda function.) I've basically stopped asking ANY coding questions in the initial interview, instead favoring reviewing their GitHub before a phone interview then a short web app project after the phone interview. (Call up this no-auth-required API, display results in a sortable table. Here are a list of requirements.) That lambda question seems kind of bad because it's an optional language feature that experienced developers have mixed opinions on. I think that things like PEP8 knowledge and using list comprehensions, context managers, etc. are okayish metrics to gauge experience by, but asking them about these things isn't nearly as good as just looking at their code to see what they write.
|
# ? Feb 9, 2022 17:29 |
|
Thanks for the replies! It sounds like what I'm doing now is probably the best course of action. Skip the language feature quiz, look at code they publish and look code they write to fulfill requirements if they make it to the coding test. Nice too because it gives me far more examples to compare against.
|
# ? Feb 9, 2022 17:36 |
|
Maybe ask them about what they think advanced python features are. That could give you a good reference point on their experience so can explore follow up questions.
|
# ? Feb 9, 2022 17:40 |
|
Howdy Python goons, I figure there's a bit of overlap of people interested in this thread seeing as a lot of ML libraries are in Python, so I've made a (probably terrible) thread to talk about AI and machine learning here: https://forums.somethingawful.com/showthread.php?threadid=3993118
|
# ? Feb 9, 2022 17:42 |
|
Protocol7 posted:Howdy Python goons, I figure there's a bit of overlap of people interested in this thread seeing as a lot of ML libraries are in Python, so I've made a (probably terrible) thread to talk about AI and machine learning here: Neat, the Scientific/Math(s) Computing thread wasn't really a great practical ML chat thread. Suggestion for your OP, include a "how to get started" resource. I suggest FastAI, specifically linking the first course so they can just dive in. https://course.fast.ai/videos/?lesson=1
|
# ? Feb 9, 2022 17:45 |
There’s a slow burning data science thread, where we talk about practising ML, amongst other things.
|
|
# ? Feb 9, 2022 18:00 |
|
D34THROW posted:I just discovered "pickling" Python objects - and that SQLAlchemy supports a Pickle field type natively. Is it worth trying to pickle objects or is it better just to serialize them as JSON and **dict them back to objects when loading them? As with most things in programming, the answer is it depends. Pickling can be pretty useful in simple scenarios just because it's very easy to do. You won't see it used much in larger systems for two reasons: (1) if your pickled object contains _any_ user (i.e. untrusted) input it's a massive security risk, and (2) it's considerably less useful if you're working with anything that's not the specific Python version you pickled it in (say, another language or DB). If your data has any business value, the latter will almost always be the case, so you'll see a lot more serialization to JSON or Protobufs or Thrift in the wild.
|
# ? Feb 9, 2022 18:32 |
|
Frankly, I'm already regretting the structure this project took. It's growing into something much more similar to what I was trying to accomplish in Access two years ago but I didn't set it up that way - what I had envisioned back then was a supplement to our ERP software to track customer interactions and take care of calculating common things as painlessly as possible. I'm already storing data as JSON in one report, might as well continue to do so.
|
# ? Feb 9, 2022 19:13 |
|
Yeah I would basically always lean JSON for any real business application. I would also lean JSON for personal projects, just because Pickle has done nothing but bite me and everyone I know.
|
# ? Feb 9, 2022 19:25 |
|
Pie Colony posted:You won't see it used much in larger systems for two reasons: (1) if your pickled object contains _any_ user (i.e. untrusted) input it's a massive security risk, There's an interesting blog post on this that I love, not primarily because it talks about this, but because it's titled "Dangerous Pickles" https://intoli.com/blog/dangerous-pickles/
|
# ? Feb 9, 2022 19:29 |
Pickle was described to me on Day One of a friend evangelizing Python to me many years ago as one of the best features of the language and best reasons to use it. Every other time since then that I've ever encountered it, it was in the context of some security bulletin urgently warning you to move away from it or some package requiring you to redesign your app to not use it.
|
|
# ? Feb 9, 2022 19:31 |
|
Boy, nested conditional and looping statements make this poo poo a lot easier than trying to eyeball-parse a ton of block-level conditionals and loops.Python code:
|
# ? Feb 9, 2022 22:29 |
|
Data Graham posted:Pickle was described to me on Day One of a friend evangelizing Python to me many years ago as one of the best features of the language and best reasons to use it. Its pretty convenient way to save and load some deep learning models.
|
# ? Feb 9, 2022 23:58 |
|
CarForumPoster posted:Thread, I interview Python coders. I often get new grads who list themselves as Python (Advanced) in their skills section. 100% of them I am disappointed with - what don’t you like about python? - what do you wish it did better? - why don’t you think it does things that way? - what makes you use it in spite of that? I think if someone can answer those in a cogent and consistent way they’re pretty much what you’re looking for without needing language trivia
|
# ? Feb 10, 2022 01:29 |
|
For those curious, it looks like the Python for Data Analysis book written by the creator of pandas is coming out with a 3rd edition. The first 6 chapters are available for free at the moment with the book being in Early Access: https://wesmckinney.com/book/
|
# ? Feb 10, 2022 04:03 |
|
D34THROW posted:
Eep! range is a built-in, you should use a different variable name there
|
# ? Feb 10, 2022 06:48 |
|
I've got a Django site that wants to run tasks and I could use celery/redis-queue/whatever but these jobs are calling containers that are also Python but don't know about Django. So I want to dispatch a task from Django to one of these artifacts and give it configuration. Need something to spawn these (could be containers, could be FaaS) and then when they're done, report back with status and data. Logging would also be great. Does anyone have strategies for this? I feel like I'm circling around something but need a topic keyword to research.
|
# ? Feb 10, 2022 14:02 |
|
cinci zoo sniper posted:I’m sorry for not being too helpful here as this goes way past my preferred amount of community tech support, but k8s is just a deployment option, one paired well with containerised development environment. If this is not a horizontally scaled service, and a micro service mesh instead (which it sounds it is, as you care about having a zoo of interpreters), you should just do one repo per service and avoid inflicting upon yourself Google problems for no clear reason. Also just wanted to say thanks, this was definitely the way and now I have new problems (see above) but progressing well. Right now decoupling the system of record from the workers is an annoying price to pay but it will pay off as the other services continue to expand.
|
# ? Feb 10, 2022 14:05 |
Hed posted:I've got a Django site that wants to run tasks and I could use celery/redis-queue/whatever but these jobs are calling containers that are also Python but don't know about Django. So I want to dispatch a task from Django to one of these artifacts and give it configuration. Need something to spawn these (could be containers, could be FaaS) and then when they're done, report back with status and data. Logging would also be great. Can you reformulate what your jobs are doing in different words? “Calling containers” is quite vague.
|
|
# ? Feb 10, 2022 14:22 |
|
Sure! These tasks call a job to go scrape a forum, or thread, or threadpage. So the container would start up, read configuration (from injected environment, injected config file, http call??), then use that to call functions in the container, do the work, then eventually the work needs to go to S3, and status needs to get back to the Django site somehow. Logs can go to anywhere, I’d just like to be able to look at them when things go wrong.
|
# ? Feb 10, 2022 14:34 |
|
|
# ? May 15, 2024 16:20 |
This sounds lightweight enough that I would consider it throwing into serverless event processing, e.g., AWS Lambda, if my budget permits.
|
|
# ? Feb 10, 2022 14:44 |