Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Popete
Oct 6, 2009

This will make sure you don't suggest to the KDz
That he should grow greens instead of crushing on MCs

Grimey Drawer
Question on how best to handle shared data between two python modules. I have a class that holds state information within a dict and I have 2 modules using this class to read and modify the dict data but right now they each have their own separate objects of this class so the state data within the dict isn't shared. I'd like there to be one shared object of this class that each module can access. What is the Pythonic way of doing this?

Adbot
ADBOT LOVES YOU

D34THROW
Jan 29, 2012

RETAIL RETAIL LISTEN TO ME BITCH ABOUT RETAIL
:rant:
I just discovered "pickling" Python objects - and that SQLAlchemy supports a Pickle field type natively. Is it worth trying to pickle objects or is it better just to serialize them as JSON and **dict them back to objects when loading them?

12 rats tied together
Sep 7, 2006

Epsilon Plus posted:

How should I handle a situation in which code wants to create an instance of a class, but shouldn't because it's being passed the wrong data?
<snip>

Let's say a really needs to be an integer - if we pass it something that can't be converted into an integer, I need to make sure an instance of SampleClass isn't created.

There are a lot of ways to tackle this, but probably the easiest would be
Python code:
def __init__(self, a: int, b: any, c: any):
    self.a = int(a)
[...]
which will raise a ValueError if whatever input can't be cast to int. You might want to write a try_cast_to_int() helper if you need more complicated logic than that.

If you have the time, it would be better to use something like https://github.com/python-desert/desert to generate serialization schemas and validate against that schema while constructing. I assume you're doing some kind of network stuff here as well, but if you're not, you can also use a static type checker to make sure that your codebase isn't passing anything that isn't an integer to this function.

Popete posted:

Question on how best to handle shared data between two python modules. I have a class that holds state information within a dict and I have 2 modules using this class to read and modify the dict data but right now they each have their own separate objects of this class so the state data within the dict isn't shared. I'd like there to be one shared object of this class that each module can access. What is the Pythonic way of doing this?
You can put it in a module and import it, just create a "shared.py" or something and then "from shared import mydict".

death cob for cutie
Dec 30, 2006

dwarves won't delve no more
too much splatting down on Zot:4

12 rats tied together posted:

There are a lot of ways to tackle this, but probably the easiest would be
Python code:
def __init__(self, a: int, b: any, c: any):
    self.a = int(a)
[...]
which will raise a ValueError if whatever input can't be cast to int. You might want to write a try_cast_to_int() helper if you need more complicated logic than that.

Isn't the instance of the object still created at that point, though? Wouldn't I just end up with an uninitialized/partially initialized object of that class? I mean if it raises an exception right then and brings down the whole program it's not really an issue anymore per se, but I don't want the entire program to fall over irrecoverably over it.

D34THROW
Jan 29, 2012

RETAIL RETAIL LISTEN TO ME BITCH ABOUT RETAIL
:rant:
Crossposting from the SQL questions thread.

I'm working on a Flask app to improve workflows for me and as coding/webdev practice, using SQLite3/SQLAlchemy as the backend.

I have a table color with the fields id, name, abbrev. This is intended to be a lookup table and something that I could pull off easily in Access.

I also have a table storm_panel_header with fields id, customer, job_no, header_color, sill_color, line_items where line_items is the one side of a one-to-many relationship with StormPanelLines.

Where I am running into trouble is defining the type of relationship between color and storm_panel_header. Both header_color and sill_color need to get their values from color.id, but color is also intended, like I said, as a lookup table.

That is to say, storm_panel_header will not be the one and only table referring to color by color.id.

I cannot figure out how to assign the foreign key and backref between the two and it's pissing me off.


EDIT: If I specify a ForeignKey relationship on storm_panel_header.header_color and
storm_panel_header.sill_color but no backref in color, does that essentially make it a one-way lookup? I don't ever foresee a situation in which I am going to need to look up what items across several tables have which color.

D34THROW fucked around with this message at 21:53 on Feb 8, 2022

cinci zoo sniper
Mar 15, 2013




I’m not sure I understand the problem. You don’t know how to generate foreign keys in code, or how to model your problem in SQL?

The latter is trivial, header_color_id and still_color_id are foreign keys referencing color.id. The real problem your schema is line_items. Storing an array there can cause giant problems for certain query patterns, and the way to prevent that is to have separate table called line_item with fields id, storm_panel_header_id, and storm_line_item_id, where the latter two are foreign keys of id fields of respective tables.

cinci zoo sniper fucked around with this message at 22:01 on Feb 8, 2022

Phobeste
Apr 9, 2006

never, like, count out Touchdown Tom, man

Epsilon Plus posted:

Isn't the instance of the object still created at that point, though? Wouldn't I just end up with an uninitialized/partially initialized object of that class? I mean if it raises an exception right then and brings down the whole program it's not really an issue anymore per se, but I don't want the entire program to fall over irrecoverably over it.

Python object initialization isn't done until __init__ is complete, so this

Python code:

class A:
   def __init__(self): raise Exception("i shan't be doing this while my hair is wet")
try:
   a = A()
except Exception:
   print("drat, the hair was wet")
print(a)

will result in a NameError at print(a), since the raise was during the lhs of the class creation.

In 99.99% of cases you shouldn't use __new__ and if you're in the .01% you'll know it, guido.

If you're worried about saving memory, hey it's a gc'd language it'll go away eventually probably

D34THROW
Jan 29, 2012

RETAIL RETAIL LISTEN TO ME BITCH ABOUT RETAIL
:rant:

cinci zoo sniper posted:

I’m not sure I understand the problem. You don’t know how to generate foreign keys in code, or how to model your problem in SQL?

The latter is trivial, header_color_id and still_color_id are foreign keys referencing color.id. The real problem your schema is line_items. Storing an array there can cause giant problems for certain query patterns, and the way to prevent that is to have separate table called line_item with fields id, storm_panel_header_id, and storm_line_item_id, where the latter two are foreign keys of id fields of respective tables.

Oh, storm_panel_lines is going to have fields for enough data to compute the particulars for a report, id, header_id, width, height, trapped, top_attach, bottom_attach. That's enough for the actual class object to do the lookups and math.

I think I was overcomplicating things. Any individual color doesn't need to know what line items have its id so I don't think I need to backref() that table to its "children".


EDIT: line_items holds the relationship of the header (parent) to its children in StormPanelLines - the one side of the one-to-many of header-to-lines.

D34THROW fucked around with this message at 22:15 on Feb 8, 2022

cinci zoo sniper
Mar 15, 2013




If lines has header reference, I have no idea why you want to reference lines in header at all.

QuarkJets
Sep 8, 2008

D34THROW posted:

I just discovered "pickling" Python objects - and that SQLAlchemy supports a Pickle field type natively. Is it worth trying to pickle objects or is it better just to serialize them as JSON and **dict them back to objects when loading them?

If you have a dict of basic types then that already maps so well to json that you should just write a json, but if you have some complicated object that you need to store and load later then you could pickle it. I don't like writing entire objects to disk and would rather convert some of the information going in/out, but I'm sure there are situations I haven't imagined where I would be like "o yeah pickling this makes sense"

QuarkJets
Sep 8, 2008

Popete posted:

Question on how best to handle shared data between two python modules. I have a class that holds state information within a dict and I have 2 modules using this class to read and modify the dict data but right now they each have their own separate objects of this class so the state data within the dict isn't shared. I'd like there to be one shared object of this class that each module can access. What is the Pythonic way of doing this?

What you're looking for is a class attribute, but what you're describing (a module secretly modifying shared data being used by another module, but without being called directly?) is frustrating to debug and has poor readability. You should define functions that mutate data in a clear and concise way. Have one of these modules pass an instance of the class to the other module for mutation

QuarkJets
Sep 8, 2008

Epsilon Plus posted:

Isn't the instance of the object still created at that point, though? Wouldn't I just end up with an uninitialized/partially initialized object of that class? I mean if it raises an exception right then and brings down the whole program it's not really an issue anymore per se, but I don't want the entire program to fall over irrecoverably over it.

Raising an exception doesn't mean that the program falls over irrecoverably, the exception just has to be caught. Raising an exception for bad inputs is the most commonly used and also best way to do what you want to do

death cob for cutie
Dec 30, 2006

dwarves won't delve no more
too much splatting down on Zot:4

Phobeste posted:

Python object initialization isn't done until __init__ is complete, so this


poo poo, I thought it was done when __new__ was complete... which would rely on __init__ being complete...

... why did I need to ask this question, again? I could have sworn it creates an instance, then uses __init__ to populate it afterwards.

Phobeste
Apr 9, 2006

never, like, count out Touchdown Tom, man

Epsilon Plus posted:

poo poo, I thought it was done when __new__ was complete... which would rely on __init__ being complete...

... why did I need to ask this question, again? I could have sworn it creates an instance, then uses __init__ to populate it afterwards.

i'm pretty sure that's how it works internally but when you're creating an object they both get done at the same sequence point relative to the creator invocation so if either fails the result won't get bound

BeastOfExmoor
Aug 19, 2003

I will be gone, but not forever.
Ok, I've been trying to figure this out for a bit and I'm stumped. I'm writing a test for the simple timer function below. I'd like to patch time.sleep() so that countdown_timer is called in the test it basically runs as a nothing function and takes essentially no time at all so that I can test larger numbers to ensure the counter is working without actually waiting the timer to sleep. I'm looking through explanations of how to do this, but I'm a bit stumped?

Python code:
def countdown_timer(total_seconds: int, leading_text: str = "", timer_speed: float = 1, countdown_increment: int = 1) -> None:
    remaining_seconds = int(total_seconds)
    while(remaining_seconds >= 1):
        hours, seconds = divmod(remaining_seconds, (60*60))
        minutes, seconds = divmod(seconds, 60)
        timer_string = f"{hours:02d}:{minutes:02d}:{seconds:02d}"  # Format 01:02:03
        print(f"{leading_text}{timer_string}", end="\r")
        time.sleep(countdown_increment * timer_speed)
        remaining_seconds -= countdown_increment
    print("Done!", " " * 20, "\n", end="\r")


@pytest.mark.parametrize(
    "total_seconds, expected_time, first_print_string",
    [
        (1, 1, "00:00:01"),
        (2, 2, "00:00:02"),
        (3, 3, "00:00:03")
    ]
    )
def test_countdown_timer(total_seconds: int, expected_time, first_print_string):
    start_time = datetime.datetime.now()
    with mock.patch('sys.stdout', new=io.StringIO()) as fake_stdout:
        countdown_timer(total_seconds)
    end_time = datetime.datetime.now()
    delta = end_time - start_time
    assert round(delta.total_seconds()) == total_seconds
    assert fake_stdout.getvalue().split("\r")[0] == first_print_string
[/code


edit: Figured it out. Not sure why that was so hard. I think the passed through function tripped me up.

[code=Python]
@pytest.mark.parametrize(
    "total_seconds, expected_time, first_print_string",
    [
        (1, 1, "00:00:01"),
        (2, 2, "00:00:02"),
        (3, 3, "00:00:03")
    ]
    )
@mock.patch('time.sleep')
def test_countdown_timer(mock_sleep, total_seconds: int, expected_time, first_print_string):
    start_time = datetime.datetime.now()
    with mock.patch('sys.stdout', new=io.StringIO()) as fake_stdout:
        countdown_timer(total_seconds)
    end_time = datetime.datetime.now()
    # delta = end_time - start_time
    # assert round(delta.total_seconds()) == total_seconds
    assert fake_stdout.getvalue().split("\r")[0] == first_print_string

BeastOfExmoor fucked around with this message at 08:49 on Feb 9, 2022

QuarkJets
Sep 8, 2008

Something like this should work:

Python code:
from unittest.mock import patch

@patch('time.sleep', return_value=None)
@pytest.mark.parametrize(...)
def test_countdown_timer():
    # Test logic
You can use patch() as a decorator or as a context manager if you don't want the mock to be used for the entire test

CarForumPoster
Jun 26, 2013

⚡POWER⚡
Thread, I interview Python coders. I often get new grads who list themselves as Python (Advanced) in their skills section. 100% of them I am disappointed with what I feel like are basic questions about how Python is used here in practice, which means it might be me. (For example, describe how you might use and anonymous or lambda function.) I've basically stopped asking ANY coding questions in the initial interview, instead favoring reviewing their GitHub before a phone interview then a short web app project after the phone interview. (Call up this no-auth-required API, display results in a sortable table. Here are a list of requirements.)

Google results for "python interview questions" have quiz-like questions ABOUT the language that I feel aren't great for seeing who can learn quickly and has spent 1000s of hours to become an advanced level practitioner.

Are there any good questions you'd ask someone who describes themselves as a Python expert that wouldn't cut out people who simply used Python in a different way?

I've so far been very happy with the quality of my hires, but always looking to get better.

CarForumPoster fucked around with this message at 16:01 on Feb 9, 2022

necrotic
Aug 2, 2005
I owe my brother big time for this!
Edit: I need to refresh

cinci zoo sniper
Mar 15, 2013




CarForumPoster posted:

Thread, I interview Python coders. I often get new grads who list themselves as Python (Advanced) in their skills section. 100% of them I am disappointed with what I feel like are basic questions about how Python is used here in practice, which means it might be me. (For example, describe how you might use and anonymous or lambda function.) I've basically stopped asking ANY coding questions in the initial interview, instead favoring reviewing their GitHub before a phone interview then a short web app project after the phone interview. (Call up this no-auth-required API, display results in a sortable table. Here are a list of requirements.)

Google results for "python interview questions" have quiz-like questions ABOUT the language that I feel aren't great for seeing who can learn quickly and has spent 1000s of hours to become an advanced level practitioner.

Are there any good questions you'd ask someone who describes themselves as a Python expert that wouldn't cut out people who simply used Python in a different way?

I've so far been very happy with the quality of my hires, but always looking to get better.

I personally don’t believe that asking language trivia on an interview is all that useful, but if you must insist, this is a better filter than just mauling the applicant flow with a barbed LeetCode, I guess.

My 5 minute hot take on possible questions for someone aiming for a generic senior Python role:

1) When are context managers useful?
2) What are decorators in Python? When would you use one?
3) What are advantages and disadvantages of pickling?
4) What is GIL? What can you do in a situation where it is a performance bottleneck?

QuarkJets
Sep 8, 2008

CarForumPoster posted:

Thread, I interview Python coders. I often get new grads who list themselves as Python (Advanced) in their skills section. 100% of them I am disappointed with what I feel like are basic questions about how Python is used here in practice, which means it might be me. (For example, describe how you might use and anonymous or lambda function.) I've basically stopped asking ANY coding questions in the initial interview, instead favoring reviewing their GitHub before a phone interview then a short web app project after the phone interview. (Call up this no-auth-required API, display results in a sortable table. Here are a list of requirements.)

Google results for "python interview questions" have quiz-like questions ABOUT the language that I feel aren't great for seeing who can learn quickly and has spent 1000s of hours to become an advanced level practitioner.

Are there any good questions you'd ask someone who describes themselves as a Python expert that wouldn't cut out people who simply used Python in a different way?

I've so far been very happy with the quality of my hires, but always looking to get better.

That lambda question seems kind of bad because it's an optional language feature that experienced developers have mixed opinions on.

I think that things like PEP8 knowledge and using list comprehensions, context managers, etc. are okayish metrics to gauge experience by, but asking them about these things isn't nearly as good as just looking at their code to see what they write.

CarForumPoster
Jun 26, 2013

⚡POWER⚡
Thanks for the replies! It sounds like what I'm doing now is probably the best course of action. Skip the language feature quiz, look at code they publish and look code they write to fulfill requirements if they make it to the coding test. Nice too because it gives me far more examples to compare against.

a dingus
Mar 22, 2008

Rhetorical questions only
Fun Shoe
Maybe ask them about what they think advanced python features are. That could give you a good reference point on their experience so can explore follow up questions.

Macichne Leainig
Jul 26, 2012

by VG
Howdy Python goons, I figure there's a bit of overlap of people interested in this thread seeing as a lot of ML libraries are in Python, so I've made a (probably terrible) thread to talk about AI and machine learning here:

https://forums.somethingawful.com/showthread.php?threadid=3993118

CarForumPoster
Jun 26, 2013

⚡POWER⚡

Protocol7 posted:

Howdy Python goons, I figure there's a bit of overlap of people interested in this thread seeing as a lot of ML libraries are in Python, so I've made a (probably terrible) thread to talk about AI and machine learning here:

https://forums.somethingawful.com/showthread.php?threadid=3993118

Neat, the Scientific/Math(s) Computing thread wasn't really a great practical ML chat thread. Suggestion for your OP, include a "how to get started" resource. I suggest FastAI, specifically linking the first course so they can just dive in. https://course.fast.ai/videos/?lesson=1

cinci zoo sniper
Mar 15, 2013




There’s a slow burning data science thread, where we talk about practising ML, amongst other things.

Pie Colony
Dec 8, 2006
I AM SUCH A FUCKUP THAT I CAN'T EVEN POST IN AN E/N THREAD I STARTED

D34THROW posted:

I just discovered "pickling" Python objects - and that SQLAlchemy supports a Pickle field type natively. Is it worth trying to pickle objects or is it better just to serialize them as JSON and **dict them back to objects when loading them?

As with most things in programming, the answer is it depends. Pickling can be pretty useful in simple scenarios just because it's very easy to do. You won't see it used much in larger systems for two reasons: (1) if your pickled object contains _any_ user (i.e. untrusted) input it's a massive security risk, and (2) it's considerably less useful if you're working with anything that's not the specific Python version you pickled it in (say, another language or DB). If your data has any business value, the latter will almost always be the case, so you'll see a lot more serialization to JSON or Protobufs or Thrift in the wild.

D34THROW
Jan 29, 2012

RETAIL RETAIL LISTEN TO ME BITCH ABOUT RETAIL
:rant:
Frankly, I'm already regretting the structure this project took. It's growing into something much more similar to what I was trying to accomplish in Access two years ago but I didn't set it up that way - what I had envisioned back then was a supplement to our ERP software to track customer interactions and take care of calculating common things as painlessly as possible. I'm already storing data as JSON in one report, might as well continue to do so.

The March Hare
Oct 15, 2006

Je rêve d'un
Wayne's World 3
Buglord
Yeah I would basically always lean JSON for any real business application. I would also lean JSON for personal projects, just because Pickle has done nothing but bite me and everyone I know.

death cob for cutie
Dec 30, 2006

dwarves won't delve no more
too much splatting down on Zot:4

Pie Colony posted:

You won't see it used much in larger systems for two reasons: (1) if your pickled object contains _any_ user (i.e. untrusted) input it's a massive security risk,

There's an interesting blog post on this that I love, not primarily because it talks about this, but because it's titled "Dangerous Pickles" https://intoli.com/blog/dangerous-pickles/

Data Graham
Dec 28, 2009

📈📊🍪😋



Pickle was described to me on Day One of a friend evangelizing Python to me many years ago as one of the best features of the language and best reasons to use it.

Every other time since then that I've ever encountered it, it was in the context of some security bulletin urgently warning you to move away from it or some package requiring you to redesign your app to not use it.

D34THROW
Jan 29, 2012

RETAIL RETAIL LISTEN TO ME BITCH ABOUT RETAIL
:rant:
Boy, nested conditional and looping statements make this poo poo a lot easier than trying to eyeball-parse a ton of block-level conditionals and loops.

Python code:
# Calculate the DLO adders, then the DLO based on direction.
dlo_add_header = mount.query.get(attach_top).dlo_add
dlo_add_sill = mount.query.get(attach_bottom).dlo_add
dlo_target = ((self.height + dlo_add_header + dlo_add_sill) if
    direction == 1 else (self.width + dlo_add_header + dlo_add_sill))
# Figure out which direction to calculate the coverage.
target_coverage = self.width if direction == 1 else self.height
# Figure out how many panels.
range = np.arange(14.25, 224.25, 6)
self.coverage = min(c for c in range if c > target_coverage)
self.panels = ((self.coverage - 14.25) / 12 ) + 1
self.track_length = math.ceil(self.coverage)

CarForumPoster
Jun 26, 2013

⚡POWER⚡

Data Graham posted:

Pickle was described to me on Day One of a friend evangelizing Python to me many years ago as one of the best features of the language and best reasons to use it.

Every other time since then that I've ever encountered it, it was in the context of some security bulletin urgently warning you to move away from it or some package requiring you to redesign your app to not use it.

Its pretty convenient way to save and load some deep learning models.

Phobeste
Apr 9, 2006

never, like, count out Touchdown Tom, man

CarForumPoster posted:

Thread, I interview Python coders. I often get new grads who list themselves as Python (Advanced) in their skills section. 100% of them I am disappointed with

- what don’t you like about python?
- what do you wish it did better?
- why don’t you think it does things that way?
- what makes you use it in spite of that?

I think if someone can answer those in a cogent and consistent way they’re pretty much what you’re looking for without needing language trivia

Hughmoris
Apr 21, 2007
Let's go to the abyss!
For those curious, it looks like the Python for Data Analysis book written by the creator of pandas is coming out with a 3rd edition. The first 6 chapters are available for free at the moment with the book being in Early Access:

https://wesmckinney.com/book/

QuarkJets
Sep 8, 2008

D34THROW posted:

Python code:
# Figure out how many panels.
range = np.arange(14.25, 224.25, 6)

Eep! range is a built-in, you should use a different variable name there

Hed
Mar 31, 2004

Fun Shoe
I've got a Django site that wants to run tasks and I could use celery/redis-queue/whatever but these jobs are calling containers that are also Python but don't know about Django. So I want to dispatch a task from Django to one of these artifacts and give it configuration. Need something to spawn these (could be containers, could be FaaS) and then when they're done, report back with status and data. Logging would also be great.

Does anyone have strategies for this? I feel like I'm circling around something but need a topic keyword to research.

Hed
Mar 31, 2004

Fun Shoe

cinci zoo sniper posted:

I’m sorry for not being too helpful here as this goes way past my preferred amount of community tech support, but k8s is just a deployment option, one paired well with containerised development environment. If this is not a horizontally scaled service, and a micro service mesh instead (which it sounds it is, as you care about having a zoo of interpreters), you should just do one repo per service and avoid inflicting upon yourself Google problems for no clear reason.

Also just wanted to say thanks, this was definitely the way and now I have new problems (see above) but progressing well. Right now decoupling the system of record from the workers is an annoying price to pay but it will pay off as the other services continue to expand.

cinci zoo sniper
Mar 15, 2013




Hed posted:

I've got a Django site that wants to run tasks and I could use celery/redis-queue/whatever but these jobs are calling containers that are also Python but don't know about Django. So I want to dispatch a task from Django to one of these artifacts and give it configuration. Need something to spawn these (could be containers, could be FaaS) and then when they're done, report back with status and data. Logging would also be great.

Does anyone have strategies for this? I feel like I'm circling around something but need a topic keyword to research.

Can you reformulate what your jobs are doing in different words? “Calling containers” is quite vague.

Hed
Mar 31, 2004

Fun Shoe
Sure! These tasks call a job to go scrape a forum, or thread, or threadpage. So the container would start up, read configuration (from injected environment, injected config file, http call??), then use that to call functions in the container, do the work, then eventually the work needs to go to S3, and status needs to get back to the Django site somehow. Logs can go to anywhere, I’d just like to be able to look at them when things go wrong.

Adbot
ADBOT LOVES YOU

cinci zoo sniper
Mar 15, 2013




This sounds lightweight enough that I would consider it throwing into serverless event processing, e.g., AWS Lambda, if my budget permits.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply