|
Falcon2001 posted:Speaking of plots and datavis/etc, I'm curious how people would approach this problem: I'm not sure why you would need an arbitrary datatype, the whole problem is constrained by some integer arrays and 1 pair of integer coordinates (the offset of the starting point for the array). 0 if a beacon is not present, 1 if it is. Another array for sensors, another array for "beacon cannot be here" positions. I think that's it If you want to get fancy you can define your own numpy dtype. 3 integer arrays or 1 array where the datatype is 3 integers, two ways of representing the same problem.
|
#
?
Oct 19, 2023 16:23
|
|
|
|
| # ? May 13, 2024 22:02 |
|
|
QuarkJets posted:I'm not sure why you would need an arbitrary datatype, the whole problem is constrained by some integer arrays and 1 pair of integer coordinates (the offset of the starting point for the array). 0 if a beacon is not present, 1 if it is. Another array for sensors, another array for "beacon cannot be here" positions. I think that's it Yeah, the arbitrary datatype part was more for other problems, as this one absolutely can be handled with a single integer even, as there's no real overlap between the three states. And honestly this problem in particular is probably really solved via math around distances to various points. But it does sound like dataframes are maybe a decent place to keep iterating on. Zoracle Zed posted:a dict or defaultdict with a tuple of ints as the key type would be my first try Yep, exactly. I'd love to be able to visualize these setups quickly like they do in the explanations (which I've sort of got working with my PIL solution) here's what one of their example explainers looks like: code:StumblyWumbly posted:Have you tried using DataFrame.iloc calls with Pandas? That can let you have weirdo indices and also check neighbors I think from reading that since it's using integer based indexing it falls into the whole negative values are interpreted as underflow issue, but I'll dig into it. .loc has been fine so far for testing. So it sounds like my dataframe approach isn't insane, at least for ones where visualizing it is reasonable (there was a falling sand puzzle previously that definitely benefited from that), so I'll stick with it for now. I think any time it's super slow is an example of a time where populating the whole map probably isn't a good idea anyway.
|
|
#
?
Oct 19, 2023 19:06
|
|
|
Falcon2001 posted:So it sounds like my dataframe approach isn't insane, at least for ones where visualizing it is reasonable (there was a falling sand puzzle previously that definitely benefited from that), so I'll stick with it for now. I think any time it's super slow is an example of a time where populating the whole map probably isn't a good idea anyway. You are doing it like a multiindex and not just a tuple valued index right?
|
|
#
?
Oct 19, 2023 19:17
|
|
|
Chin Strap posted:You are doing it like a multiindex and not just a tuple valued index right? Not a tuple valued index no, just a standard 2d dataframe with custom indices - here's the generation code. Python code:
|
|
#
?
Oct 19, 2023 20:19
|
|
|
Falcon2001 posted:Not a tuple valued index no, just a standard 2d dataframe with custom indices - here's the generation code. Oh I missed you only wanted 2d. This is fine, or you could make a thin wrapper around a numpy ndarray with an offset for each axis stored so that when you do `x[i, j]` it actually calls `x[i + x_min, j + y_min]` maybe. Could be faster for algorithms.
|
|
#
?
Oct 19, 2023 20:27
|
|
|
Falcon2001 posted:Yep, exactly. I'd love to be able to visualize these setups quickly like they do in the explanations (which I've sort of got working with my PIL solution) here's what one of their example explainers looks like: the ascii grid isn't too bad: code:
|
|
#
?
Oct 19, 2023 22:00
|
|
|
Zoracle Zed posted:the ascii grid isn't too bad: Yeah, that's what I started with, and I've got a helper function that creates an ASCII grid from my dataframes, but once you get past a certain size it's hard to read, and it's not like text editors are well made for that sort of horizontal/vertical scrolling/etc.
|
|
#
?
Oct 20, 2023 01:31
|
|
|
Related to all this pandas discussion: what's y'all's take on the future of pandas now that polars exists? I know that polars doesn't yet do everything that pandas does (though I'm not an expert on the details here), but it does a hell of a lot, and with a nicer API and vastly more speed. I'll be curious to see polars' impact over time given that it seems to be surging in popularity. Personally, I've switched over to polars when I can, though I often still bang something out in pandas when I need something quick and the execution speed differences don't really matter.
|
|
#
?
Oct 20, 2023 11:40
|
|
|
Zugzwang posted:Related to all this pandas discussion: what's y'all's take on the future of pandas now that polars exists? I know that polars doesn't yet do everything that pandas does (though I'm not an expert on the details here), but it does a hell of a lot, and with a nicer API and vastly more speed. I'll be curious to see polars' impact over time given that it seems to be surging in popularity. I think "nicer API" is a bit of a stretch. Haven't used it because at Google we don't really have approved Rust support yet (cross language compilation with all the custom things we do takes a lot of support and it is still definitely alpha) but it looks more like a thin layer over SQL style operations than the pandas library. Never felt super pythonic to me to have to write something like (stolen from reddit) code:Also there is something to be said for a much more mature package with more books and other packages that work with it. If the dataframe interchange protocols get off the ground there is less of an argument here. And you can always convert directly to a pandas dataframe for those api boundaries I guess (but I'm guessing that doing that too often will cause slowdowns too). But I can't deny the speed. And there is definitely something to the fact that the API has been able to be built without an extreme amount of baggage it carries around. Pandas is trying its best to make things more consistent and streamlined but deprecation and removal take forever. Copy-on-write in Pandas 2 is probably going to become the default, in which case by Pandas 3 they are liable to remove all inplace operations and make things immutable the way Polars does which would be a big step forward. I know some people don't like the chaining syntax of Polars, but that's how modern Pandas should be written more often than not anyway (and again how you see it with tidyverse stuff in R so it is the style I am used to). I am a Pandas contributor though so part of my Polars reluctance may also just be clinging to what I know. I certainly would be giving polars a shot if I could, but would probably wind up going with pandas for non-performance critical parts because of the syntax.
|
|
#
?
Oct 20, 2023 14:53
|
|
|
I'm a total python noob. I'm trying to get more in-depth info on list comprehensions, it's one of the python things that I'm having some issue coding and using, since I don't think there is anything similar in Java and my instructor insists that we use them for everything. I was wondering if there was any resources I could be pointed to that goes over, for example, how to use a dict of lambda functions to sort a list of lists using a one-liner list comprehension. I already got the MetaSnake python book and it's been very helpful, so thank you to whoever suggested it. BUUNNI fucked around with this message at 01:58 on Oct 21, 2023 |
|
#
?
Oct 21, 2023 01:54
|
|
|
Chin Strap posted:I think "nicer API" is a bit of a stretch. Haven't used it because at Google we don't really have approved Rust support yet (cross language compilation with all the custom things we do takes a lot of support and it is still definitely alpha) but it looks more like a thin layer over SQL style operations than the pandas library. Never felt super pythonic to me to have to write something like (stolen from reddit)
|
|
#
?
Oct 21, 2023 02:12
|
|
|
BUUNNI posted:I'm a total python noob. I'm trying to get more in-depth info on list comprehensions, it's one of the python things that I'm having some issue coding and using, since I don't think there is anything similar in Java and my instructor insists that we use them for everything. List compressions are pretty intuitive, maybe you could just ask about the details you're having trouble with, they can't be that ba... quote:
... okay What your instructor wants you to learn is that if you can write a for loop that appends entries to a list, then you can turn that into a list comprehension. It doesn't matter whether that's a nested for loop or includes conditions and break points, you can always refactor down to a list comprehension. The simplest way to figure this out is to write some dumb nested for loops that append to a list first, then build up the list comprehension starting from the innermost loop. This also lets you prove to yourself that your answer is correct, because you should feel confident about the nested append version The biggest challenge is writing something that looks clean, people like to think that they're writing clever "one liners" that are really 10 lines long once you apply a character limit, and that look like incomprehensible bullshit. The key is to offload logic to other functions, which can also be easily tested. You can use these same functions in a dumb nested for loop or a comprehension, so it's easy to think your way through what's happening
|
|
#
?
Oct 21, 2023 02:41
|
|
|
Two tips for writing list comprehension: include a comment about what you're trying to do, and use descriptive variable names, even if it triples the number of characters. List comprehension are often obvious only while you are writing them. Also it seems like the kind of thing chatgpt would be good at, but I have not tested that
|
|
#
?
Oct 21, 2023 13:06
|
|
|
StumblyWumbly posted:Also it seems like the kind of thing chatgpt would be good at, but I have not tested that Its better than me, but thats not hard.
|
|
#
?
Oct 21, 2023 13:14
|
|
|
RealPython has a good article on list comprehensions: https://realpython.com/list-comprehension-python/ (Their stuff is usually good, FYI)
|
|
#
?
Oct 21, 2023 13:47
|
|
|
StumblyWumbly posted:Also it seems like the kind of thing chatgpt would be good at, but I have not tested that We're not allowed to use AI for our examinations  StumblyWumbly posted:Two tips for writing list comprehension: include a comment about what you're trying to do, and use descriptive variable names, even if it triples the number of characters. Thank you for this btw BUUNNI fucked around with this message at 03:42 on Oct 24, 2023 |
|
#
?
Oct 21, 2023 22:27
|
|
|
Falcon2001 posted:Why not use UUIDv4 instead of generating this weird custom hashing thing? It's basically guaranteed to not collide and v4 is generated from a seed. The collision chance of UUIDv4 is essentially zero - one explanation put it at 'generate a million UUIDv4s a day for 100 years and you'll have a 50% chance of a single duplicate.' Just wanted to update real quick--I reached a point where I was more or less at v1 and started looking at optimizations and the first thing I did was replace generate_p_index with a simple uuid.uuid4() call, and it's over 3x faster now lmao Much appreciated!
|
|
#
?
Oct 27, 2023 18:40
|
|
|
quote:https://code.djangoproject.com/ticket/373 Guess I won't be using Django then.
|
|
#
?
Nov 2, 2023 02:22
|
|
|
TasogareNoKagi posted:Guess I won't be using Django then. Its actually coming reasonably soon though. Theres literally work happening right now to get CompositeField in. Its apparently fairly tricky though and its been a work in progress for quite a few months.
|
|
#
?
Nov 4, 2023 09:42
|
|
|
TasogareNoKagi posted:Guess I won't be using Django then. Why would you use composite keys (idk poo poo about DBs)
|
|
#
?
Nov 4, 2023 11:58
|
|
|
Sometimes you want a natural key that isn’t an auto-increment because you have to regularly import new data destructively over the old data and FK relationships have to keep working; and sometimes the only way to ensure row uniqueness that way is combining multiple columns together. That’s what I’m dealing with right now, it’s a legacy ETL process and SQLalchemy supports it (though not very intuitively), but Django looks like a non-starter. We’re probably going to use uuid pk’s instead anyway though, the composite key thing makes my brain hurt
|
|
#
?
Nov 4, 2023 12:27
|
|
|
Even then you almost certainly want a synthetic primary key plus a separate unique constraint on the set of columns you think should be unique.
|
|
#
?
Nov 4, 2023 14:09
|
|
|
duck monster posted:Its actually coming reasonably soon though. Theres literally work happening right now to get CompositeField in. Its apparently fairly tricky though and its been a work in progress for quite a few months. "Soon" isn't "now", though. Though it may be moot; PHB learned that I was using Python and demanded I rewrite it all in Typescript  Jabor posted:Even then you almost certainly want a synthetic primary key plus a separate unique constraint on the set of columns you think should be unique. That was the suggested "solution" I saw while reading up on this holy war in DB design. In my view, it only solves the "foreign keys are  issue , while adding additional complexity and storage space. Having the additional unique constraint means the DB is still going to create and maintain a composite index on those columns, so it isn't solving anything, just hiding it.
|
|
#
?
Nov 5, 2023 01:56
|
|
|
The things it solves are major issues that you run into when you're trying to operate a system in reality instead of as a homework assignment that will be thrown away after it's been graded. Notably, you now have a realistic path for changing your constraint when (not if, when) it turns out that your initial domain modelling wasn't correct. Also you're actually saving space rather than spending more of it, since other tables referencing your synthetic key are smaller than if they had to reference your entire composite key.
|
|
#
?
Nov 5, 2023 04:07
|
|
|
TasogareNoKagi posted:"Soon" isn't "now", though. Though it may be moot; PHB learned that I was using Python and demanded I rewrite it all in Typescript Yeah I've had this happen twice before (In once case the demand was to rewrite in PHP). In both cases it resulted in sharply written resignation letters (And in the case of the PHP one, the whole loving team followed me out the door. GG shittyboss. That company doesnt exist no more. One day people might learn that 30 years of experience matters.)
|
|
#
?
Nov 6, 2023 02:37
|
|
|
How would you store a string describing a series of key:values pairs in a concise human editable way, that can be safely parsed into a native python dict ? I'd rather not rely on eval() for exemple I wrote a function with re.search and splits and it doesn't feel elegant at all.
|
|
#
?
Nov 6, 2023 18:18
|
|
|
Would CSV work? fart,1 butt,2 etc,3
|
|
#
?
Nov 6, 2023 18:31
|
|
|
I should add: it's going to be on a single line with few values, keys and values are ASCII letters, it can look like this: "name:steph,position=shelter".
|
|
#
?
Nov 6, 2023 18:36
|
|
|
I always lean towards existing standards so that I can use built in tools and make it easier for other systems to interact with it, so have you considered something like JSON?
|
|
#
?
Nov 6, 2023 18:39
|
|
|
1. pick 2 delimiters: one for key+value and one for element. split, split, assign. 2. use a markup language. yaml toml and json all allow mappings to be expressed in a single line. toml comes included in python but yaml is likely the most human readable. make sure to read the documentation for your yaml library because it does some type inference that might be surprising to you.
|
|
#
?
Nov 6, 2023 18:46
|
|
|
Just use JSON.
|
|
#
?
Nov 6, 2023 18:53
|
|
|
Json is ok but it needs doubles quotes which add some visual noise. It probably makes the most sense though. Thanks for the pointers.
|
|
#
?
Nov 6, 2023 20:03
|
|
|
rowkey bilbao posted:How would you store a string describing a series of key:values pairs in a concise human editable way, that can be safely parsed into a native python dict ? I'd rather not rely on eval() for exemple I would use json
|
|
#
?
Nov 7, 2023 04:45
|
|
|
Does human editable mean 'editable by a programmer' (use something standard like everyone else has said) or 'editable by Bob from Marketing who needs explicit very friendly error messages on typos'?
|
|
#
?
Nov 7, 2023 06:12
|
|
|
Foxfire_ posted:Does human editable mean 'editable by a programmer' (use something standard like everyone else has said) or 'editable by Bob from Marketing who needs explicit very friendly error messages on typos'? code:
|
|
#
?
Nov 7, 2023 16:21
|
|
|
The Fool posted:
Then the fun begins when you have to reingest that XLSX 
|
|
#
?
Nov 7, 2023 17:19
|
|
|
Macichne Leainig posted:Then the fun begins when you have to reingest that XLSX I think Pandas reads it alright, but I'm sure there's complications.
|
|
#
?
Nov 7, 2023 17:21
|
|
|
There is a read_excel but let me tell you about Bob from Marketing's inability to enter data consistently I'd rather just use a CSV if I was doing anything tabular but even then that's not foolproof (and necessitates that the Bobs from Marketing know how to perform such an operation to begin with) I feel like I'm having deja vu, didn't we have a discussion in this very thread a few pages back about ingesting data? Or is my brain finally loving with me too much
|
|
#
?
Nov 7, 2023 17:32
|
|
|
Data ingestion is a common problem and CSV is always the solution
|
|
#
?
Nov 7, 2023 17:47
|
|
|
|
| # ? May 13, 2024 22:02 |
|
|
spiritual bypass posted:Data ingestion is a common problem and CSV is always the solution Speaking of which, and this isn't really a Python question, but is there an open source CSV editor that has some of the functionality of excel without all the...overhead? It'd be nice to find something I can use to just muck around quickly with tabular data without having to constantly be like 'no, don't accidentally make it an xlsx, stop formatting it' etc - not that it's a super big problem or anything.
|
|
#
?
Nov 7, 2023 18:03
|
|