pokeyman posted:

Anytime someone says the format is "CSV", save yourself a headache and ask "which CSV?"

canis minor posted:

This is what I wanted to hear, thank you very much! It's a good thing that I'm leaving this place and won't be dealing with engineering this (as, tracking who/what/when edited the CSVs, or the state of how many CSV are there is an information that needs to be tracked in a DB )

Bruegels Fuckbooks posted:

ultrafilter posted:

You can store the CSVs from a given run, which makes reproducing/debugging significantly easier. If you're going to talk to the database directly, you'll need some way to recreate the state of the database at the time a model was run to reproduce the run.

quote:

The line comment is useful when you want to restate what the next line does. Here's an idiomatic example:

code:

// adds one to counter counter++;;;

block comment

code:

/* this is a block comment */

The block comment is useful when a comment is long, like explaining some implementation:

code:

/* The International Enterprise™ Association only certifies code with a block comment that exceeds three lines, so this comment guarantees our future certification. */

On top of these, Enterprise™ adds:

copyright comment

code:

/© This code is property of ACME™ studios 2017. ©/

Every Enterprise™ program must begin with a copyright notice, else it will not compile and fail with an UnexpectedNonDisruptiveOpenSourceException error.

It's customary to cover any non trivial implementation in Enterprise™ with a copyright (and a comment). On top of that add an NDA comment (see below).

NDA comment

code:

/© This following code implements a "Web Dropdown Menu", copyright number 9283F3. ©/ 
/NDA The following code can only be read if you signed NDA 375-1. If you happen to read it by mistake, send a written letter to our legal department with two attached copies immediately. NDA/

boo_radley posted:

Enterprise: the language spec

https://github.com/joaomilho/Enterprise/blob/master/README.md

canis minor posted:

Ok, yes, though you could achieve the same thing as having previous states as snapshots of the DB, just like copying the CSVs . I imagine as well, that you'd only want such behaviour if you're tweaking your algorithm (unless you're tweaking it all the time).
The person that will be implementing this though was handwaving about performance, and using CSV as a standard in Python machine learning (I don't know anything about the subject), and how you should always be using numpy + pandas + CSV over anything else, even local sqlite - hence the "why" question.

Don't get me wrong, I do appreciate all the points made - I'd happily accept if I was told "I want to open this dataset in Excel, see if it makes sense, maybe run some function on it that I know how to write in Excel" or "I want to operate on CSVs because that's what I'm accustomed to and seeing text files makes me happy".

xtal posted:

If you used a local database or SQLite they would probably be approximately equivalent in terms of speed, but much, much more powerful. You could even query the database for the results as CSV and pipe that in to your program instead and the slowdown would be minimal. Whether or not the entire dataset fits in memory is a big difference, but orthogonal to the question of CSV vs. RDBMS.

QuarkJets posted:

Being able to open the file in Notepad and read through its contents is not a portability feature, it's an interpretability feature (and a pointless one at that)

QuarkJets posted:

Also CSV is not a good format for portability, as your files are needlessly huge.

boo_radley posted:

Enterprise: the language spec

https://github.com/joaomilho/Enterprise/blob/master/README.md

Carbon dioxide posted:

This link made my Linux Firefox tab crash.

redleader posted:

Why do you think that's pointless? I've found it fantastically useful to be able to open and gently caress around with CSVs in any old editor.

canis minor posted:

Ok, yes, though you could achieve the same thing as having previous states as snapshots of the DB, just like copying the CSVs . I imagine as well, that you'd only want such behaviour if you're tweaking your algorithm (unless you're tweaking it all the time).

canis minor posted:

So, it's the matter of transferring datasets, which I guess, yes, there's a point there. I guess, if it run on the same box, would there still be much difference? (let's assume that data is read-only for machine learning algorithms, but will indeed change at any point after the algorithms are run)

QuarkJets posted:

my two cents, he's kind of wrong, CSV is common but HDF5 is common too and way superior, for all reasons except "can be opened with notepad"

Anyone who says "CSV should obviously be the standard" for what often amounts to GB or sometimes even TB of numerical data is a fool

JawnV6 posted:

Or if for some wild reason the data in the DB was subject to change and the person wanted something reproducible. Like the previous page had someone with this issue:

canis minor posted:

I'm sorry but, I don't understand the point you're making - I'd assume that if you'd want something reproducible you'd create a dump, run algorithm. Let's say day passes, data changes - so you create a dump, run your algorithm again. If you want to get back to previous day you load the previous dump. You'd have reproducible snapshots and using a CSV in that scenario isn't better than using a DB - you can still do that using DBs.

I don't know if at any point you can do without the previous snapshots, but I'd hope at certain point your algorithm works and doesn't need to be fine tuned to provided data so you can do without previous snapshots (I'm sure that I'm wrong ).

JawnV6 posted:

Okay, did the folks you're actually working with bring this use case to your attention, or was it just my shitpost? Do you naturally set up a DB to provide point-in-time snapshots on queries like this? You didn't bother to distinguish between "the data I pulled has changed" and "the data I pulled has not changed, but there is additional data relevant to the query" so I'm not that confident you understand what you're doing over there.

"I ran my algo yesterday and got an amazing model! Then I ran my training again today and the model is crap. Can you help me debug this?" - This is the guy who's going to tell you CSV's would have saved him, who wasn't sophisticated enough to put things into DB-snapshot terms because frankly that seems like your job.

canis minor posted:

Will keep that in mind - I'd think that if you want to look at the data, you'd want to perform some operations (aggregate, filter, etc.) which I don't see how CSV is suited for - you're essentially using Excel to do stuff to that dataset, so you end up with "SQL", so why not use SQL to begin with. At my place they bought water cooled PC to deal with Excel parsing through this CSV, which is just (but again, not something I deal with)

canis minor posted:

Ok, yes, though you could achieve the same thing as having previous states as snapshots of the DB, just like copying the CSVs . I imagine as well, that you'd only want such behaviour if you're tweaking your algorithm (unless you're tweaking it all the time).
The person that will be implementing this though was handwaving about performance, and using CSV as a standard in Python machine learning (I don't know anything about the subject), and how you should always be using numpy + pandas + CSV over anything else, even local sqlite - hence the "why" question.

Don't get me wrong, I do appreciate all the points made - I'd happily accept if I was told "I want to open this dataset in Excel, see if it makes sense, maybe run some function on it that I know how to write in Excel" or "I want to operate on CSVs because that's what I'm accustomed to and seeing text files makes me happy".

canis minor posted:

Ok, yes, though you could achieve the same thing as having previous states as snapshots of the DB, just like copying the CSVs . I imagine as well, that you'd only want such behaviour if you're tweaking your algorithm (unless you're tweaking it all the time).

return0 posted:

It’s totally reasonable to abstract the model training implementation from the DB or API by using an exported immutable snapshot as its input. I’d personally use JSON lines or any other splittable format, but whatever.

Connecting directly to the DB is bad. Keeping snapshot archives as a DB dump is bad. Having splittable input is good.

You’ve characterised their response to your objection as hand waving. Perhaps that’s because this stuff is fairly standard good practice, and they didn’t understand where you were coming from.

ryde posted:

You are aways tweaking it. We want to "re-train" ML algorithms periodically on new data, in part because we're detecting behavior and it changes over time.

Likewise you may have to re-train a previous version of your model. A lot of ML tools, such as scikit, expect you to just serialize the whole object graph using stuff like Java serialization/Dill and then deserialize it for prod. This can obviously break with library versions or code change. So you might have to rebuild your serialized object graph at some point, and you want the original data around.

If you're lucky you have a somewhat implementation independent format for describing the models and retraining isn't necessarily a thing, but you also have a bunch of transformations before you even feed data into the model and you may have to retrain when *that* changes too.

Edit: to put it another way, its not handwaving. Versioning ML algorithms involves building a version over the input data, transforms on the data, the training code, and library support code. Hard to version data that you don't snapshot.

Edit2: There's also no reason it has to be CSV, except for tooling. We would prefer something like Parquet or Avro, but tooling can be limited so end up with JSON more often than not.