Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
seance snacks
Mar 30, 2007

Anyone had any experience using ChatGPT within the context of a Flask application?

I've inherited a small web app which acts as a customer portal. Pretty simple stuff, lots of APIs integrated to query stuff like Salesforce or Smartsheet and put it in a little table so clients can log in and see some info of their contracts with us, project progress, generate little matplotlib graphs, etc.

I haven't messed with ChatGPT yet beyond making an account and reading some documentation. My understanding is that the prompt space is limited so I'm thinking that I'd want to give it the context of the web app, it's libraries and API's used but still have space to describe a feature I'd like to add or maybe troubleshoot an existing feature?

If something like that is doable then I'm thinking I could use it to "draw the lines" of a new feature and then I'd manually "color it in" with relevant variable names, etc.

Adbot
ADBOT LOVES YOU

AfricanBootyShine
Jan 9, 2006

Snake wins.

Falcon2001 posted:

Can you add some more details about what you're trying to change about the data? Is it actually breaking CSV convention or just weird? etc etc.
Often the instrumentation will drop in a few extra lines before or after the CSV starts with date/time/protocol info. Another instrument has headers for each sample and then a second set of headers below for each time point. Then if there are replicates, it's three sets of nested headers prior to the data.

I have written a few scripts to import data generated by some instruments, but in terms of time saved it's often faster to manually clean up the CSV by hand/export as an excel file etc. That's what other people do but I'd like to automate it as a little project to stay up on my python. I don't want to spend *too* much time on it as working with this data is a relatively small part of my job.

Re: the XML files, I've tried using every xml import library I could find and they all choke. There's a library in R that was written to import an older version of these data files, which I might just have to convert.

Sorry for the lack of detail- I'm currently on holiday and once I get back I can provide some concrete examples.

CarForumPoster
Jun 26, 2013

⚡POWER⚡

seance snacks posted:

Anyone had any experience using ChatGPT within the context of a Flask application?

I've inherited a small web app which acts as a customer portal. Pretty simple stuff, lots of APIs integrated to query stuff like Salesforce or Smartsheet and put it in a little table so clients can log in and see some info of their contracts with us, project progress, generate little matplotlib graphs, etc.

I haven't messed with ChatGPT yet beyond making an account and reading some documentation. My understanding is that the prompt space is limited so I'm thinking that I'd want to give it the context of the web app, it's libraries and API's used but still have space to describe a feature I'd like to add or maybe troubleshoot an existing feature?

If something like that is doable then I'm thinking I could use it to "draw the lines" of a new feature and then I'd manually "color it in" with relevant variable names, etc.

Just try it

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug

AfricanBootyShine posted:

Often the instrumentation will drop in a few extra lines before or after the CSV starts with date/time/protocol info. Another instrument has headers for each sample and then a second set of headers below for each time point. Then if there are replicates, it's three sets of nested headers prior to the data.

I have written a few scripts to import data generated by some instruments, but in terms of time saved it's often faster to manually clean up the CSV by hand/export as an excel file etc. That's what other people do but I'd like to automate it as a little project to stay up on my python. I don't want to spend *too* much time on it as working with this data is a relatively small part of my job.

Re: the XML files, I've tried using every xml import library I could find and they all choke. There's a library in R that was written to import an older version of these data files, which I might just have to convert.

Sorry for the lack of detail- I'm currently on holiday and once I get back I can provide some concrete examples.

All programming is an exercise in 'time saved vs time spent' - https://xkcd.com/1205/ is the classic example, but I'd say that sort of work would bother me personally.

FWIW: At least based on what you're saying I'd say that a single script with multiple 'options' as arguments/etc is probably a reasonable thing to build once and reuse forever. (If the filenames are consistent enough to derive instrument from then you might be able to get away with no args but like, args good.) If the output is unified, then you have the opportunity to setup a simple dataclass and then write functions to turn X into Y.

None of the stuff you've mentioned so far would be terribly complex to munge into a single thing using basic readlines() and loops/etc, the trickiest part is trying to detect those things, so if you can expect them, it gets a lot simpler.


Yeah, if you're at a place that lets you ChatGPT, just play around with it. No harm in trying it out for a while and watching some videos/etc.

StumblyWumbly
Sep 12, 2007

Batmanticore!

Falcon2001 posted:

All programming is an exercise in 'time saved vs time spent' - https://xkcd.com/1205/ is the classic example, but I'd say that sort of work would bother me personally.
I'd also add that time spent learning Python vs cleaning CSVs will pay future dividends, as will time spent thinking through the cleanup process enough to get someone else (or some _thing_ else) to do the work for you.

QuarkJets
Sep 8, 2008

It also sounds like there are other people working with the same data who also have to clean it up by hand, maybe they'd appreciate a tool that could help do that job

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug
Honestly poo poo like that is 100% why I got into development. Nothing better than smoothly automating some lovely process. One of the first things I did on my last team was move this fourteen step excel file munging process (that had started to take like, 8 minutes to load thanks to a bazillion rows) to be Pandas-driven instead; even outputting styled spreadsheets like it used to. Whole thing also runs end to end in like 5 seconds.

Just my favorite thing ever. I don't want to cure cancer, I just want to go around and help automate little lovely things that suck.

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams
Working at a large public research university, I know that I enjoy automating the lovely things, but as a bonus I'm also making it slightly easier for someone else to cure cancer. Best of both worlds!

boofhead
Feb 18, 2021

I'm a backend dev for a consultancy and about 80% of my workload is building unspecced ambitious new tools for internal stakeholders with infinitive complexity requirements that either get "parked indefinitely" once I ask them to test the finished product or it gets released and then immediately forgotten about and abandoned because they're already desperate for the next highest priority project that's going to 'solve everything', or people demanding I spend a month building a similarly unspecced automation pipeline to save a cumulative 4 hours of manual data entry per year, which will also get abandoned or outdated 2 weeks after it's released

And before you tell me to push back on it, my company refuses to hire product managers, I've worked as one in my last two jobs and that experience was part of the reason to hire me, except when I try to push back on "everything is highest priority all the time and needs to be developed simultaneously" nonsense or try to manage expectations or even just ask for actual requirements, my boss's boss steps in and basically demands to know why I'm being antagonistic or shirking work

So it turns out that they're happy with 80% of my work taking ages and then going straight down the drain so long as I just say "sure, I'll work on it soon" and they don't have to do any actual managing

Lol. Lmao. Just guffaws all round

On the upside, because there's literally no oversight and I work from home, there's almost zero pressure and I have a very chill working life, and they even authorised me to go down to 80% time next month (with pay cut unfortunately) and I get to teach myself a bunch of poo poo that would otherwise be impossible if my employer was any shade of competent. So it's a mixed bag

duck monster
Dec 15, 2004

boofhead posted:

I'm a backend dev for a consultancy and about 80% of my workload is building unspecced ambitious new tools for internal stakeholders with infinitive complexity requirements that either get "parked indefinitely" once I ask them to test the finished product or it gets released and then immediately forgotten about and abandoned because they're already desperate for the next highest priority project that's going to 'solve everything', or people demanding I spend a month building a similarly unspecced automation pipeline to save a cumulative 4 hours of manual data entry per year, which will also get abandoned or outdated 2 weeks after it's released

And before you tell me to push back on it, my company refuses to hire product managers, I've worked as one in my last two jobs and that experience was part of the reason to hire me, except when I try to push back on "everything is highest priority all the time and needs to be developed simultaneously" nonsense or try to manage expectations or even just ask for actual requirements, my boss's boss steps in and basically demands to know why I'm being antagonistic or shirking work

So it turns out that they're happy with 80% of my work taking ages and then going straight down the drain so long as I just say "sure, I'll work on it soon" and they don't have to do any actual managing

Lol. Lmao. Just guffaws all round

On the upside, because there's literally no oversight and I work from home, there's almost zero pressure and I have a very chill working life, and they even authorised me to go down to 80% time next month (with pay cut unfortunately) and I get to teach myself a bunch of poo poo that would otherwise be impossible if my employer was any shade of competent. So it's a mixed bag

Yeah this is pretty much my job too. I keep begging the boss to stop hiring teeth grindingly stupid PHP idiots and hire a project manager , and he just ignores it. So projects keep going off the rails.

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug

duck monster posted:

Yeah this is pretty much my job too. I keep begging the boss to stop hiring teeth grindingly stupid PHP idiots and hire a project manager , and he just ignores it. So projects keep going off the rails.

Hilariously this is actually not far from my current state, although we're all generally working on stuff that gets used and is useful, but there's no PM layer to translate and we're basically just really lucky that management's like 'uhhh yeah write good things'. So the end results is...good? Although loving chaotic.

On Python topics, are there any good python-specific approaches to writing tests? Not like 'how am pytest' docs, but more of a thing where 'What tests should you be writing and why'.

I've developed some strong opinions in my time as a dev, but I'm also pretty sure some of those opinions are wrong; I watched a talk linked here about the failures of TDD that I thought was really interesting, but when I was trying to implement this someone brought up that creates tests that are testing huge swaths of code at once, which makes test failures complex to debug and also has the problem of removing tests that work double duty as 'here's how this shared function works in practice' that can be pretty helpful.

A lot of stuff I've found is very fully OO-oriented, but the code my team tends to write is a combination of OO and functional programming, where we use a mix of classes where appropriate and atomic functions for shared behaviors that are easy to represent in this way. It's something I have found to be really nice about Python, but a lot of the testing advice for Python tends to just be 'here's how to use unit test!'.

If it's covered somewhere in Fluent Python that's one more reason to buy the drat book I guess.

QuarkJets
Sep 8, 2008

There are a zillion development textbooks that probably have a better answer but at the end of the day good Python testing methodology is the same as good testing methodology in other languages. Since pytest is fully compatible with unittest classes but adds a lot of extra features, you should probably use pytest for unit and workflow testing. If you do that then you're already ahead of like 90% of other software developers

Falcon2001 posted:

I've developed some strong opinions in my time as a dev, but I'm also pretty sure some of those opinions are wrong; I watched a talk linked here about the failures of TDD that I thought was really interesting, but when I was trying to implement this someone brought up that creates tests that are testing huge swaths of code at once, which makes test failures complex to debug and also has the problem of removing tests that work double duty as 'here's how this shared function works in practice' that can be pretty helpful.

Was this a talk about behavior-driven development (BDD)? BDD still needs unit testing and workflow testing, if that speaker was saying otherwise then they're offering bad advice but I think they were probably just focusing on the idea of collaborating with non-developers for writing scenarios, which is cool and good but shouldn't be considered a requirement for every scenario. Ideal behavior-driven development includes scenarios that only describe units of functionality; under the hood that's just a unit test that you call with `behave` instead of `pytest` or something else. BDD is meant to help developers figure out what tests to write and how to write them, so it may help you identify some good system-level tests that you wouldn't have otherwise considered

Hed
Mar 31, 2004

Fun Shoe
I love pytest so freaking much. For some smaller internal packages I feel like it's a big dependency to add. Do you have any examples (e.g. good packages) that handle a "production distribution" but still have tests available if you want it with optional dependencies?

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug

QuarkJets posted:

There are a zillion development textbooks that probably have a better answer but at the end of the day good Python testing methodology is the same as good testing methodology in other languages. Since pytest is fully compatible with unittest classes but adds a lot of extra features, you should probably use pytest for unit and workflow testing. If you do that then you're already ahead of like 90% of other software developers

Was this a talk about behavior-driven development (BDD)? BDD still needs unit testing and workflow testing, if that speaker was saying otherwise then they're offering bad advice but I think they were probably just focusing on the idea of collaborating with non-developers for writing scenarios, which is cool and good but shouldn't be considered a requirement for every scenario. Ideal behavior-driven development includes scenarios that only describe units of functionality; under the hood that's just a unit test that you call with `behave` instead of `pytest` or something else. BDD is meant to help developers figure out what tests to write and how to write them, so it may help you identify some good system-level tests that you wouldn't have otherwise considered

FWIW: I could be wrong about all of this, so I'd appreciate pointing out where:

The argument I saw was not exactly BDD, but summed up best as 'Don't test every individual function you write, test things your code does, which generally correlate with Classes in most OO languages'. The trap I've seen (and fallen into myself) is that once you write something even remotely complex, you end up decomposing it into individual functions, and then end up writing unit tests for every individual function.

This adds a huge amount of friction to refactoring efforts and prevents the sort of Red / Green / Refactor flow that TDD is meant to encourage.

https://www.youtube.com/watch?v=EZ05e7EMOLM is the full talk if you want to check my work (and I might have taken away the wrong idea from it, to be fair). I might just buy Test Driven Development and give it a read since the big argument from this talk was 'most people didn't read the book and blindly implemented it from wikipedia'.

A simple example would be a Calculator class; your calculator does a certain number of things, but to accomplish those things you might want to for example, call wolfram alpha for a particular thing, or use some standardized formatting methods; but if you write a bunch of detail-specific tests you can't refactor easily because every refactor requires a bunch of reworking of all your tests.

If you write your tests to check 'Can I add, can I subtract, etc', how you do those things doesn't matter, as long as the input/output is accurate. This lets you quickly refactor the 'middle bits' as much as you want, and rely on the tests to tell you that you didn't break anything about the functionality, which is what really matters.

For a specific example of where it's been causing issues, I'm working with a state machine based scripting framework, where multiple solutions are coded using the same framework across my team, which enforces a certain amount of consistency. A given solution has a number of states but have a consistent entry/exit point.

So my tests basically create an instance of my solution, and when I test a given state I run the solution through the whole state machine, modifying things as necessary using pytest's parametrize decorator to account for combinations of different things. So I have one test per state of the machine, with multiple parameters, and a few one-off tests for odd side-effects. Some of those states are complex, however, and are decomposed into multiple methods, which are only used as part of the given state flow.

I ran into a couple downsides with this approach though:

1. Because you might call a given API multiple times for a given scenario and only need to modify the Xth time you call it (think safety checks), the mocking setup for those got pretty complex, with a bunch of side effects that did have to change depending on refactoring (read API calls, to be clear, not writes, which should be consistent)
2. If I want to test the final step in the workflow I need to test all the steps before it as well.
3. If I create any common functions that are shared among different solutions, I'm just relying on my specific test testing that it works, and if someone else wants to see 'hey what does this do in practice' they can't *really* look at my testing code.

I suspect upon writing this that the biggest problem is that our state machine has some limitations on which states can be run solo; if we setup a testing implementation of that state machine that let you run them independently I'd probably have a cleaner setup.

ninja edit: Obviously having at least one end to end test is probably a good idea (this solution is a single class, so it's not as though I'm testing a massive codebase in one run).

Falcon2001 fucked around with this message at 20:03 on May 11, 2023

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug

Hed posted:

I love pytest so freaking much. For some smaller internal packages I feel like it's a big dependency to add. Do you have any examples (e.g. good packages) that handle a "production distribution" but still have tests available if you want it with optional dependencies?

I believe one standard way is to use a requirements-dev.txt? But I also haven't build many packages myself and work at a company with a proprietary build system (where we just have a separate list of build/development/runtime stuff). I think in general from looking around stuff like PyPi doesn't manage test dependencies (because if you're developing it, you're generally getting it from source, which will have a requirements.txt or a requirements-dev.txt, or some other instructions for grabbing them.)

Falcon2001 fucked around with this message at 20:45 on May 11, 2023

SurgicalOntologist
Jun 17, 2004

We use extras_require['test'] in setup.py for test dependencies that don't end up in the build (neither do the tests themselves, if you package things correctly). Although I gather setup.py is deprecated and we're behind the times, I assume there's an equivalent.

If someone wants tests, rather than running from PyPi/docker/whatever build, they should pull the source. The only reason to run the tests is if you're developing anyway, assuming you can see in the CI that they passed for the current build.

QuarkJets
Sep 8, 2008

Hed posted:

I love pytest so freaking much. For some smaller internal packages I feel like it's a big dependency to add. Do you have any examples (e.g. good packages) that handle a "production distribution" but still have tests available if you want it with optional dependencies?

You can list optional dependencies in a setup.cfg or a pyproject file under a label that you've chosen, then when someone tries to pip install your package they won't get those dependencies unless they know to ask for them. So you could set up a block of extra dependencies labeled "tests" and then pip would only install those when asked: `pip install mypackage[tests]`. If you're working with the package locally (e.g. as a developer), `pip install -e .[tests]`

e: Oh beaten, yeah what SurgicalOntologist said. Instead of using a setup.py use a setup.cfg or a pyproject, this is a pip feature so they all support it

Hed
Mar 31, 2004

Fun Shoe
Thanks all, I have only done a setup.py a long time ago and as I understood things have changed.

Precambrian Video Games
Aug 19, 2002



There's a great but unfortunately named C++ header-only unit testing library doctest. It's similar to pytest but also allows writing tests alongside code, something that seems like it would be great once you get used to it. Sadly I haven't felt bold enough to try, nor have I figured out a solution to maintaining test coverage on modules with pybind11 bindings without duplication (there isn't exactly redundancy because you should test all bound functions anyway as it's easy to misconfigure bindings).

Ironically, I did not even know that the Python standard library module doctest existed until about a year ago. It offers a similar feature as C++ doctest: it "searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown." A colleague used it to test examples in tutorial docs but I wonder if it could be used for unit test coverage too.

QuarkJets
Sep 8, 2008

Hed posted:

Thanks all, I have only done a setup.py a long time ago and as I understood things have changed.

That's right, setup.py is still supported but at minimum you should switch over to using a setup.cfg (which is basically a setup.py but in a config file)

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams
Just as a few of my own cents on testing...

I've spent the last year on and off working on a FastAPI project. My general structure is that I have methods that represent the endpoints of the API (obviously) but they do very little work directly, and basically just assemble data to be sent to various "utility" functions. I could test all those utility functions directly, but instead I'm using pytest and pytest-cov to test my API endpoints, generally running through multiple scenarios to cover all the paths of that utility code.

I've done quite a bit of major refactoring under the hood, but my tests have barely changed because I'm testing the functionality of the app directly, rather than what's underneath. Not that 100% code coverage is the be-all-end-all, but I have gotten it to 100% code coverage (with a few "# pragma: no cover" comments here and there).

And it's pretty common to have something like package[dev] that will install whatever tools are needed for development and testing. Technically speaking you could do that when pulling a package from pypi, but it's really for development (in which case you'd run pip install .[dev] in your local copy of the repo).

Seventh Arrow
Jan 26, 2005

Favorite python youtube channels?

I like Indently, he always has stuff that's new and interesting to me: https://www.youtube.com/@Indently

https://www.youtube.com/watch?v=qvSjZ6AKfXQ

https://www.youtube.com/watch?v=tATFQUx0Zx0

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug
I watch a fair bit of https://www.youtube.com/c/arjancodes and find him to be pretty pleasant to listen to and interesting.

Jigsaw
Aug 14, 2008
I like https://www.youtube.com/@mCoding. Always to the point and super clear. Also he has some C++ stuff if you care about that, but it’s mostly Python.

Zugzwang
Jan 2, 2005

You have a kind of sick desperation in your laugh.


Ramrod XTreme
Seconding both ArjanCodes and mCoding. The former has a great deal of stuff on general software engineering, with examples implemented in Python. The latter is mostly but not exclusively Python (he also covers C++ stuff sometimes).

Falcon2001
Oct 10, 2004

Eat your hamburgers, Apollo.
Pillbug

Falcon2001 posted:

Bunch of testing talk

BTW related to this, I think that after talking to a few other people I've started to develop a general heuristic/rule for this, although one discussion reminded me no rule should be followed 100%, but this is more of a starting point.

For unit tests: You should test the smallest portion of your code that can or will be used independently of others. If this is too large and tests feel clunky, further break that down by analyzing what it does.

For example, when you decompose a larger function down into smaller ones, obviously the platonic ideal is that those end up being reusable but that's not always the case, and is often simply a way to clean up flow and make stuff more readable. In those cases, writing tests for each of the decomposed functions doesn't really make sense; you should just test your 'glue' function that ties them together, which probably corresponds to something your program does.

Example: My state machine function; Ideally, I'd like to test each state independent of each other. There's some complexity there since we're using Click and a state machine/etc, but I think I can make it work. However, we also have independent reusable functions that are used for 'common' stuff; these should have their own, individual tests (even if they are technically covered from a coverage perspective by my personal feature tests).

Part of the reason I'm trying to work through this is that I have folks on my team that have never written tests before and wanted advice so I'm trying to write up some basic advice.

duck monster
Dec 15, 2004

This is why you, don't let ChatGPT generate content for your website without review.

I present ;- "The goto statement in python", including information about the "comehere" statement. Not an April 1sts post.

https://www.upgrad.com/blog/goto-statement-in-python/

Well its either ChatGPT or "loving idiot". There is no goto in python.

Seventh Arrow
Jan 26, 2005

People confuse Python with BASIC all the time!

duck monster
Dec 15, 2004

Seventh Arrow posted:

People confuse Python with BASIC all the time!

Basic. never had a "comehere" statement (comehere is a command in "intercal" a sort of parody-but-real language designed intentionally to be an absolute dog to code in and generally be an incompetent-on-purpose hell experience. You have to include the optimum amount of "please, " commands in front of other commands so the language is satisfied you are being polite enough, but if too many lines have a please, it will error out because your just being condescending.

The "comehere" command in Intercal is there because "goto considered harmful", however is a thing of beauty. If it it was in basic it would look something like;-

10 comehere 20
20 print "hello"

However, this can introduce paradoxes.

10 comehere 30
20 comehere 30
30 print "hello"

(these are written in a comehere-ified basic, not intercal, cos intercal is incomprehensible

Well, which line does it go to when 30 is reached? 10, or 20?

In default Intercal, this is an error. However one variant "solved" it by making it the parallelism mechanism. It forks and returns to BOTH comehere statements. This 3 liner thus is a fork-bomb.

Also, Intercal had FORGET and IGNORE commands for flow control too (you could tell it to FORGET certain commands, or IGNORE certain lines. ABSTAIN was a more powerful IGNORE because it also abstained from raising errors making it an effective tool for writing comments.

The Intercal Manual really is a hilarious read, for something written in the 1970s

DoctorTristan
Mar 11, 2006

I would look up into your lifeless eyes and wave, like this. Can you and your associates arrange that for me, Mr. Morden?

duck monster posted:

This is why you, don't let ChatGPT generate content for your website without review.

I present ;- "The goto statement in python", including information about the "comehere" statement. Not an April 1sts post.

https://www.upgrad.com/blog/goto-statement-in-python/

Well its either ChatGPT or "loving idiot". There is no goto in python.

I’m looking forward to the future where chatGPT12 has been trained on an internet largely populated by this kind of shite

Zugzwang
Jan 2, 2005

You have a kind of sick desperation in your laugh.


Ramrod XTreme
"ChatGPT or idiot?" will now be a question I ask myself frequently.

I recently read a blog post like that one that extolled C++ as a language known for being simple and easy to learn.

CarForumPoster
Jun 26, 2013

⚡POWER⚡
All models are wrong, some models are useful.

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams
Speaking of testing, and discussion a little while back about next(), I've run into some fun bugs around that related AnyIO and respx, the testing framework for httpx.

There's some kind of issue where when the AnyIO backend is asyncio and a StopIteration error is raised, it just hangs instead of throwing the error. I was using next() a few places in my code and if the iterator was exhausted my tests would freeze (and would likely freeze in production as well). In addition, respx uses next() under the hood to find the right response to return to a mocked httpx call, and if there's no response found it just hangs. And then when it comes to coverage, apparently the coverage plugin counts an iterator as an untriggered branch if the default value or StopIteraion isn't thrown, which is... kind of weird and surprising.

Now I'm super leery of next(), even though I probably shouldn't be. And the issue in AnyIO will get fixed in the 4.0 release, whenever that happens (and whenever it percolates down to starlette and fastapi).

QuarkJets
Sep 8, 2008

Given that story I wouldn't be wary about using next(), I'd be wary of using AnyIO.

It looks like there are special pytest fixtures that need to be used when you're using anyio, I'm assuming you're already using those and this is just a bug in AnyIO? I wonder if this is related at all to a generator bug that went away in in Python3.7 and later; before that StopIteration exceptions raised inside of generators would be silently swallowed (the generator would catch the exception and conclude that it had simply become exhausted), maybe AnyIO introduced funny behavior while trying to work around that?

FISHMANPET
Mar 3, 2007

Sweet 'N Sour
Can't
Melt
Steel Beams
None of my test functions are asynchronous, which I think is what that page is describing. All my tests are synchronous, which works. What I'm testing is a FastAPI app, and testing is just using starlette's TestClient functionality, so even if my app is async, the tests don't need to be. And the AnyIO isn't a choice I made, it's what's given to me by FastAPI via starlette.

I think it might actually be an edge case with asyncio directly and its Future ability specifically, but reading too much into that my head starts to spin. It appears that maybe the "pattern" when using asyncio is to do a try/except anytime you think a StopIteration might be raised to catch it and do... literally anything but raise StopIteration.

It works fine with Trio, so what AnyIO has done is just add a wrapper when the engine is asyncio to raise a different error instead.

I'm just a simple country sysadmin, waiting for the 4.0 release of AnyIO, which has apparently been in the works since like October.

QuarkJets
Sep 8, 2008

FISHMANPET posted:

None of my test functions are asynchronous, which I think is what that page is describing. All my tests are synchronous, which works. What I'm testing is a FastAPI app, and testing is just using starlette's TestClient functionality, so even if my app is async, the tests don't need to be. And the AnyIO isn't a choice I made, it's what's given to me by FastAPI via starlette.

I think it might actually be an edge case with asyncio directly and its Future ability specifically, but reading too much into that my head starts to spin. It appears that maybe the "pattern" when using asyncio is to do a try/except anytime you think a StopIteration might be raised to catch it and do... literally anything but raise StopIteration.

It works fine with Trio, so what AnyIO has done is just add a wrapper when the engine is asyncio to raise a different error instead.

I'm just a simple country sysadmin, waiting for the 4.0 release of AnyIO, which has apparently been in the works since like October.

Oh yeah, I see the PR: https://github.com/agronholm/anyio/pull/477

Looks like someone was about to ask for a patched release: https://github.com/agronholm/anyio/issues/475
But then they're like "nevermind I'll just monkeypatch the fix in!" Geeze, poster, maybe someone else could have benefitted from that release

Seventh Arrow
Jan 26, 2005

I tried discussing the below mCoding video with ChatGPT and although my explanation wasn't perfect, its response was...interesting.

https://www.youtube.com/watch?v=rdlQzhP71pQ

I've heard it said that in python, lists somehow "remember" operations that were done on them. do you know anything about this?

quote:

Yes, you are referring to the concept of "list comprehension" in Python. List comprehensions provide a concise way to create lists based on existing lists or other iterable objects. They can also be seen as a form of functional programming in Python.

When you use list comprehension, Python "remembers" the operations performed on the original list and applies them to create a new list. Let me show you an example to illustrate this:

code:
original_list = [1, 2, 3, 4, 5]
new_list = [item * 2 for item in original_list]
print(new_list)
Output:

code:
[2, 4, 6, 8, 10]
In the above example, the original list [1, 2, 3, 4, 5] is iterated over, and each element is multiplied by 2. The resulting list is [2, 4, 6, 8, 10]. Here, Python remembers the multiplication operation applied to each item in the list and applies it to generate the new list.

List comprehensions are a powerful feature in Python and can include conditions and nested loops for more complex transformations. They provide a concise and expressive way to manipulate lists and create new ones based on existing data.

(after watching the video)

so apparently it doesn't have anything to do with list comprehensions.

it has to do with sys.getsizeof() returning different values for lists that should return the same values. for example, [0, 0, 0] returns a value of 120, whereas [0] * 3 returns 80 in memory.

from what I can tell, this is because in the python engine, sys.getsizeof is using "allocated" in "PyListObject..." instead of "ob_size" in "PyVarObject...", as well as _gc_next (8 bytes) and _gc_prev (8 bytes), and some other values. This is why an empty list will take up 56 bytes, according to sys.getsizeof(). There's more to it, but that's the gist of it.

quote:

I apologize for the confusion earlier. You are correct that the behavior you described is related to the memory allocation of lists in Python, rather than list comprehensions.

In Python, lists are dynamic and can grow or shrink as needed. When you create a list using [0, 0, 0], Python allocates memory for each individual element in the list, resulting in a larger overall size. Each 0 takes up a certain amount of memory, and the list stores references to these objects.

On the other hand, when you use the multiplication operator * to create a list like [0] * 3, Python optimizes the memory allocation by creating a single object and storing references to it multiple times. In this case, there is only one 0 object in memory, and the list contains three references to it. This optimization reduces the overall memory usage.

The sys.getsizeof() function you mentioned returns the size of the object in memory. However, it's important to note that the value returned by sys.getsizeof() may not always reflect the true memory usage of an object due to various internal optimizations and memory allocation strategies employed by the Python interpreter.

Additionally, the _gc_next and _gc_prev attributes you mentioned are related to garbage collection and are not directly responsible for the difference in memory usage between the two list creation methods.

It's worth mentioning that while understanding the memory allocation details can be helpful in certain situations, it's generally recommended to focus on writing clear and efficient code, rather than relying on specific memory optimizations unless necessary. Python's memory management and optimization mechanisms handle many of these details automatically, allowing you to focus on the logic and readability of your code.

I...ok.

(I'm not faulting it for reaching for the "list comprehensions" answer, because my first question was kind of vague. But its second answer seems to comically go off the rails after a bit.)

Jabor
Jul 16, 2010

#1 Loser at SpaceChem
You seem to be expecting the chatbot to actually know things and I'm really not sure why you have that expectation?

Data Graham
Dec 28, 2009

📈📊🍪😋



Someone in another thread mentioned

Python code:
a = "wtf"
b = "wtf"
a is b
> True

a = "wtf!"
b = "wtf!"
a is b
> False
wtf?

Adbot
ADBOT LOVES YOU

QuarkJets
Sep 8, 2008

Data Graham posted:

Someone in another thread mentioned

Python code:
a = "wtf"
b = "wtf"
a is b
> True

a = "wtf!"
b = "wtf!"
a is b
> False
wtf?

It'll blow your mind even more to know that, starting from a fresh session:

Python code:
a = "wtf!"
b = "wtf!"
a is b
> False
I don't care enough about this piece of trivia to figure out the rules behind whether python decides to use an already-existing string literal or to create a new one, it doesn't really matter imo

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply