|
Ideally, you’d be able to refactor your serializers, and eliminate the combinatorial explosion.
|
#
?
Jul 15, 2023 21:16
|
|
|
|
| # ? May 15, 2024 21:39 |
|
|
The thing I don't like about dictionary dispatch is that if the method name itself is a variable, it makes it really tricky to search for it with a Ctrl+F through your code. Like I'd want to be able to search for serializer_method_1.serialize(), but if serializer_method_1 is the value returned by a dictionary or a lookup function, and you're not specifically searching for it as a key in the lookup function, you'll miss it. Might just be a bad habit of mine. And I'll grant that polymorphism can easily go way too far (I've dealt with codebases with like 16 different implementations of the same class methods in as many far-flung places and it's a nightmare). There are upsides and downsides every which where
|
|
#
?
Jul 15, 2023 21:48
|
|
|
Data Graham posted:The thing I don't like about dictionary dispatch is that if the method name itself is a variable, it makes it really tricky to search for it with a Ctrl+F through your code. Like I'd want to be able to search for serializer_method_1.serialize(), but if serializer_method_1 is the value returned by a dictionary or a lookup function, and you're not specifically searching for it as a key in the lookup function, you'll miss it. I don't understand the issue  Maybe you could sketch up an example?
|
|
#
?
Jul 15, 2023 22:15
|
|
|
I mean, it could just as easily be that I'm shooting my mouth off and outing myself as a n00b  But like, using this example writeup https://martinheinz.dev/blog/90 If I'm using PyCharm or another IDE with a global search function and I'm trying to find all the occurrences/usages of the "add" function in the codebase (let's assume the function is something less trivial in practice), I'm going to search for it by typing add( in my global search-mo-tron. But because the add function isn't being called explicitly and directly by name, but is instead being called indirectly via a dictionary lookup (functions["add"](5, 3)), my search pattern won't find it. Unless I specifically know ahead of time that the function is part of a dictionary lookup. And then maybe in the process of investigating a bug or adding a new feature involving add I'll have introduced a new bug because I missed the occurrences where it was being called via dictionary dispatch. This burned-hand attitude is the result of having had to inherit and spelunk through codebases abandoned and bequeathed to me by previous generations of developers who were no longer around to warn me of pitfalls or quirky development patterns, which might mean I'm not on the ball enough to be aware of these kinds of patterns and have a better or more reliable way to globally search, or maybe it means this is the kind of pattern that's inherently harder to maintain because it's bound to catch people by surprise if they're not the ones who wrote it in the first place. I doubt any IDE would be able to recognize it as a "usage" even in the smartest kind of introspection, too.
|
|
#
?
Jul 15, 2023 22:43
|
|
|
Data Graham posted:I mean, it could just as easily be that I'm shooting my mouth off and outing myself as a n00b I still don't understand. Why would searching for "add" or "def add" not work in this example? I copy-pasted the example into an IDE and ran a global search for "add", it seems to be working like I'd expect  If I try to rename the function it changes all of the occurrences together, including the key name in the dictionary and the line where the dispatch is used, like I'd expect 
|
|
#
?
Jul 15, 2023 23:08
|
|
|
Because it's going to also find a million other occurrences of "add" assuming the name of the function is something that's easily conflated with something else and not a trivial example. It's like if I'm searching the codebase for the name of an attribute called "enable" and there are 5 million occurrences of "enable" but only one or two that are relevant, so I have to use search string tricks to make sure I'm finding them in the right pattern. I don't know if I have a coherent answer here. I just know that it's burned me a number of times that I was relying on a certain kind of searching trick (maybe the examples in question are slipping my mind or no longer available to me) and missed some cases because I was expecting all the relevant occurrences of the function name to conform to the invocation pattern I had in mind, but the dictionary-dispatch pattern screwed me, or otherwise maybe it's just that it wasted a bunch of my time in trying to investigate my way through a bug because the function wasn't as easily searchable as what I was expecting (i.e. I didn't know the function name ahead of time, I was stepping through the code or trying to reason my way through it but the actual function came from somewhere else via a mapping and it wasn't obvious how it flowed). Data Graham fucked around with this message at 23:38 on Jul 15, 2023 |
|
#
?
Jul 15, 2023 23:35
|
|
|
It's possible that earlier IDEs couldn't find that sort of thing or whatever (especially if you're talking pre-type hinting python) but I'm also failing to understand the issue you're mentioning, because your point about generic names being a problem seems to indicate it. I also don't really use string search, I generally use my IDEs ability to find usages of a class/method instead, which in my experience has been highly effective, although I'd you add enough abstraction it might fail to find it...but I don't think a dictionary dispatch would be a problem there as long as you have type hinting etc.
|
|
#
?
Jul 16, 2023 00:05
|
|
|
Can't you make the keys enums instead of strings if this is really a problem?
|
|
#
?
Jul 16, 2023 00:10
|
|
|
eXXon posted:Can't you make the keys enums instead of strings if this is really a problem? Sometimes I go one step further and use what I like to call Enum Dispatch, using function handles as the Enum values
|
|
#
?
Jul 16, 2023 01:46
|
|
|
Oh, another note: if you want to include an else clause in your dictionary dispatch for some reason (wasn't called out in your example), you can use dict.get with a default value instead of just accessing it by key, it's a little longer, but it still remains significantly shorter code. Edit: https://martinheinz.dev/blog/90 actually contains a much cleaner way of doing this using defaultdict. You'll still have to build your dictionary somewhere, but separating the creation out means you can make it a little easier to read later. Falcon2001 fucked around with this message at 20:44 on Jul 16, 2023 |
|
#
?
Jul 16, 2023 07:32
|
|
|
wow, ok regarding my map plotting / data questions from some days ago, forget about it. I just found this site: https://www.movemap.io/explore/us ...which is new as of a year ago, has basically no traffic or hype, and does 99% of what I was gonna build myself (it's missing mosquito data but that can largely be inferred) y'all think it's using plotly under the hood? kudos to the site author
|
|
#
?
Jul 18, 2023 00:49
|
|
|
I did a quick search and the creator said on Reddit:quote:It's nextjs, react, mapping is react-simple-maps + leaflet, ui library is material-ui. https://www.reddit.com/r/leanfire/comments/trx41p/comment/i2r18u6/ If you had any questions you could probably reach out to them, almost everybody who creates something is usually pretty ecstatic to get direct feedback and to answer questions, especially if the project hasn't become overwhelmingly popular to the point where they can't keep up with the messages boofhead fucked around with this message at 08:47 on Jul 18, 2023 |
|
#
?
Jul 18, 2023 08:45
|
|
|
Been mucking around a little bit with Textual (https://textual.textualize.io/tutorial/) and I quite like it; I have basically no frontend experience so it's kind of nice to work with something that's much simpler than a lot of major JS frameworks but follows some similar principles. Here's one example App someone made - https://github.com/LeviBorodenko/gtraceroute that's pretty neat. Again, mostly proof of concept stuff and not some massive app, but I've found it reasonably easy to work with.
|
|
#
?
Jul 18, 2023 21:43
|
|
|
Falcon2001 posted:Been mucking around a little bit with Textual (https://textual.textualize.io/tutorial/) and I quite like it; I have basically no frontend experience so it's kind of nice to work with something that's much simpler than a lot of major JS frameworks but follows some similar principles. Woah. Did not know about this Textual. Thanks for the link. I am also a backend boy and this looks extremely helpful
|
|
#
?
Jul 22, 2023 18:21
|
|
|
Rookie question here: On Jupyter Notebooks, I've seen some exercises and tutorials where you can run a cell and then you can a check to see if the answer you received is correct (like on Kaggle). Setting something like that up shouldn't be too complicated, right? I'd like to create a basic notebook "students" where they can follow along, write a little code, and then have the notebook check if their answer is correct or not. Has anyone created something like this, or have a tutorial/video to follow?
|
|
#
?
Jul 24, 2023 03:41
|
|
|
Reminder: Zed Shaw's next edition of "Python the hard way" is going to be 100% about data science lol https://twitter.com/lzsthw/status/1682407572479852545 This is going to be a train wreck.
|
|
#
?
Jul 24, 2023 05:17
|
|
|
 joins seem extremely easy and intuitive to me, what could he possibly find so confusing after using them for decades? Maybe I should write a book instead
|
|
#
?
Jul 24, 2023 07:39
|
|
|
Yeah, maybe I'm missing something very obvious or very subtle, but I've always conceptualized joins as "take every pair of rows that satisfies the join condition you've specified, and if it's an outer join then also add artificial pairs corresponding to rows that would be in no such pairs", and thus far the results have been what I expected.
Bemused Observer fucked around with this message at 18:02 on Jul 24, 2023 |
|
#
?
Jul 24, 2023 13:28
|
|
|
 Seems like now I'm the one who shares experiences with some sort of tech writer. Checkmate, computer scientists.(I've barely ever used SQL or any other relational DB and I absolutely do the same thing he describes when I have to work with them.)
|
|
#
?
Jul 24, 2023 15:56
|
|
|
On the one hand I'm not readily judging people for not liking or not understanding SQL, since the language structure and ways of expression are very different from Python. On the other hand if someone sets out to write a book on systematic analysis of data and struggles with conceptualizing joins, then it's deeply worrying.
|
|
#
?
Jul 24, 2023 18:05
|
|
|
Decades, he says. Like at that point are you just inflating the usage? "I run a query once in 1998"
|
|
#
?
Jul 25, 2023 04:12
|
|
|
Is the OP still up-to-date with the best Python 101 resources?
|
|
#
?
Jul 25, 2023 04:38
|
|
|
96 spacejam posted:Is the OP still up-to-date with the best Python 101 resources? Most of it still seems reasonably fine, but you can also ask if you have questions. The thread's not particularly busy.
|
|
#
?
Jul 25, 2023 05:30
|
|
|
duck monster posted:Reminder: Zed Shaw's next edition of "Python the hard way" is going to be 100% about data science lol This could also be that close cousin of the humblebrag; the false self-effacing statement on social media made for engagement/ publicity. Like when a techbro millionaire goes on about the setbacks he suffered when he was fired from a summer job in a warehouse or when I talk about how I’m a big dumb idiot and coincidentally am just about to release a book aimed at big dumb idiots.
|
|
#
?
Jul 25, 2023 06:15
|
|
|
I think I get the reasoning, but it's weird when it becomes "I don't understand this fundamental concept in the area I'm about to teach you about". If he said he didn't understand IP addresses or the Linux file system, I'd have no problem with that
|
|
#
?
Jul 25, 2023 07:09
|
|
|
Bemused Observer posted:I think I get the reasoning, but it's weird when it becomes "I don't understand this fundamental concept in the area I'm about to teach you about". If he said he didn't understand IP addresses or the Linux file system, I'd have no problem with that I understood IPv4, but IPv6 is firmly in 'well I guess I'm choosing to not know about that.' territory for me. I even had it professionally taught to me and I still glaze over trying to parse an ipv6 address.
|
|
#
?
Jul 25, 2023 07:48
|
|
|
Bemused Observer posted:I think I get the reasoning, but it's weird when it becomes "I don't understand this fundamental concept in the area I'm about to teach you about". If he said he didn't understand IP addresses or the Linux file system, I'd have no problem with that Yeah normally those posts go ‘setback->triumph->grift’ but unless there were followup tweets to that one it looks like he forgot to do the last two lol.
|
|
#
?
Jul 25, 2023 10:51
|
|
|
I parsed it as "sure I know what a join does and how to use it, but I don't know what it DOES if you know what I mean". Like "I must understand it at a molecular level and if I don't I am a rank charlatan and a fool (this is to convince you that my book talking about it at a user level is going to be very well informed if I'm this self-effacing)" Like "I can heel-and-toe shift and four-wheel drift around cones like Ken Block but I don't KNOW how the clutch plates actually fit together and what the pedals are doing to them, oh what a hack and a fraud I am *winku*"
|
|
#
?
Jul 25, 2023 11:20
|
|
|
I mean, I also don't know what a join DOES, inasmuch as I have no idea what exactly the DB engine does to compute the results (because I'm sure it doesn't actually inspect every pair of rows one by one). My background is in mathematics, so I'm much better at conceptualizing the results than working out what exactly the computer does to calculate the result in an optimal way. But what he says is that when the output of a JOIN query isn't what he'd expected, he just randomly changes things until it works - I think not understanding the logic of joins is very worrying if you aspire to teach people about data science.
|
|
#
?
Jul 25, 2023 13:19
|
|
|
Oh god, its worse. Apparently he wrote a book on "SQL the hard way". But by his own words doesnt understand how Joins work (and in the comments gets angry at people trying to explain it to him). I dread to imagine what that book would be like, although he has form. He wrote a book on C that got him roundly mocked by the C community for not actually understanding how C works. edit: He recomends using the SELECT table1.fields, table2.fields WHERE table1.id=table2.foreignfield syntax because "joins are insanely confusing". Which tells me intuitively "gets" inner joins as a concept, but doesn't get the syntax. Thats *weird* to me, cos the INNER JOIN <table> ON <condition> syntax is so much cleaner. duck monster fucked around with this message at 13:50 on Jul 25, 2023 |
|
#
?
Jul 25, 2023 13:38
|
|
|
Lol I should stop giving would-be authorities and "experts" the benefit of the doubt
|
|
#
?
Jul 25, 2023 13:43
|
|
|
the python book he's well-known for is not particularly good either. you might even describe it as the unnecessarily hard way to learn python
|
|
#
?
Jul 25, 2023 13:43
|
|
|
I think his python book is good because after trying it twice in high school I gave up on learning to code and talked to girls instead 
|
|
#
?
Jul 25, 2023 13:54
|
|
|
Bemused Observer posted:I mean, I also don't know what a join DOES, inasmuch as I have no idea what exactly the DB engine does to compute the results (because I'm sure it doesn't actually inspect every pair of rows one by one). My background is in mathematics, so I'm much better at conceptualizing the results than working out what exactly the computer does to calculate the result in an optimal way. It actually kind of does, but with some *huge* optimizations to the process. Basically when you give it a well formed join (Beware those subqueries, they can thwart the query planner!), it builds an execution plan that it breaks up the tables into what it needs to search for to narrow down the tables for joining, then it starts matching them up. So at this point, if you've set your tables up right, you SHOULD have indexes on your ids and possibly also (it depends) foreign keys, so it can build up a faster index (ie some form of n-tree or whatever the most efficient search strategy is for the query) to do those matches. Modern databases really are a miracle of software engineering. I've seen Postgres do complex multi table joins over terabytes of data partitioned over cluster distributed tables and return results in literal nanoseconds. And poo poo, what websites like Google do makes my head swim. Its *crazy* that poo poo works at all*. *Once in my younger years a friend of mine and me spent hours on LSD making multiple search engines like AltaVista and the like simultaneously search for terms like "Fuzzy logic" and "Beef pressed into ceiling void" just to marvel at the modern spectacle of the most powerful "mainframe" databases (Hey, it was the 90s, I had no idea how these things ACTUALLY worked) in the world dedicated their resources to searching for terms of absurdity. duck monster fucked around with this message at 13:58 on Jul 25, 2023 |
|
#
?
Jul 25, 2023 13:56
|
|
|
Yeah modern query planners are a joy to behold. The lowly database is an often overlooked modern marvel. Love digging into database internals.
|
|
#
?
Jul 25, 2023 18:04
|
|
|
Oh yes, I've only recently started doing stuff with PostgresSQL, previously all my SQLing was done in BigQuery (where you have basically unlimited processing power), but I immediately realized that the query planner is cool and real an my friend (and also I realized I need to catch up on how the query optimization actually works, at least on some level)
|
|
#
?
Jul 25, 2023 18:40
|
|
|
sql in bigquery is kind of weird, the language is not as flexible as in postgres, there are queries you just can't write. that's because under the hood it is not really a rdbms, it is more like a column store built on a giant map-reduce cluster. the sql-ish interface is just a facade. and most rdbms optimization techniques don't apply to bigquery either, for example there are no indexes if you write good clean sql in bigquery you may never notice its rough edges. part of this actually has to do with google's cost model. it is possible to construct some syntactically correct sql that bigquery will refuse to run, because it is inefficient in a way that you are not being appropriately billed for. postgres will just let you run the nasty query
|
|
#
?
Jul 25, 2023 21:07
|
|
|
I know BQ is much different than other RDBMS under the hood, but what are the kind of queries it won't allow you to write? I don't think I've run into a situation where I wanted to run a query and it didn't let me express the logic I wanted; but maybe I just didn't come across the problematic scenarios.
|
|
#
?
Jul 25, 2023 21:34
|
|
|
the biggest one i remember, this was a few years ago, was: "Correlated subqueries that reference other tables are not supported unless they can be de-correlated, such as by transforming them into an efficient JOIN" postgres would have just let me write the inefficient correlated subquery, because when the postgres sql grammar says "you can put any subquery here", they mean it. whereas bigquery is more like "you can put a subset of all subqueries here, specifically the ones that fit my execution model" the other issues were more like hitting resource limits, i remember having issues calling a complex function in a select, etc
|
|
#
?
Jul 26, 2023 00:21
|
|
|
|
| # ? May 15, 2024 21:39 |
|
|
I know this is the python thread not the sql one, but in case anyone wants to deep dive on how modern dbs work, this is an amazing lecture series: https://www.youtube.com/playlist?list=PLSE8ODhjZXjYzlLMbX3cR0sxWnRM7CLFn
|
|
#
?
Jul 26, 2023 00:41
|
|