|
Subjunctive posted:Performance? Denormalization can be cool and good. I don't want to dump the schema/data here, but performance is presently an issue. As is figuring out where the hell anything is. Performance is mostly an issue because it's impossible to reason about the data rationally, so everything is done with between 3-4 and 3-4N more subqueries than necessary. Good effort though. 
|
#
?
Feb 13, 2016 20:41
|
|
|
|
| # ? May 17, 2024 18:53 |
|
|
Here's some coding horror in the prosumer products world: http://appleinsider.com/articles/16/02/12/bug-in-adobe-creative-cloud-updater-erases-root-level-mac-data Adobe pushes an update for Creative Cloud on Macs that makes the assumption that they're going to be the first folder in the root directory and that whatever comes first is okay to wipe out. Some users don't have that folder and the first one in the list for them is .DocumentRevisions-V100, which is the OS X equivalent of the volume shadow copy store on Windows.
|
|
#
?
Feb 13, 2016 21:54
|
|
|
Amberskin posted:Him: Oh, the business spec says that field can have just 10 values. I've defines 20 just to be safe. It will never exceed that, legit! This is the part where you beat them to death with the nearest fire extinguisher.
|
|
#
?
Feb 13, 2016 22:17
|
|
|
This conversation inspired me to refresh my memory of 1NF, 2NF, 3NF, BCNF, etc. I've seen a lot of stupid stuff in database table design, but I don't think I've ever seen a table that fails to satisfy 1NF in any software I've worked on. The only thing that comes to mind is certain dynamic columns intended to contain arbitrary data that's validated elsewhere, possibly including comma-separated lists of values. But I think that's basically acceptable, if a little clumsy.
|
|
#
?
Feb 13, 2016 22:25
|
|
|
LOOK I AM A TURTLE posted:This conversation inspired me to refresh my memory of 1NF, 2NF, 3NF, BCNF, etc. I've seen a lot of stupid stuff in database table design, but I don't think I've ever seen a table that fails to satisfy 1NF in any software I've worked on. The only thing that comes to mind is certain dynamic columns intended to contain arbitrary data that's validated elsewhere, possibly including comma-separated lists of values. But I think that's basically acceptable, if a little clumsy. Nah, "a list of things in one column" is a normal form violation. One way to think about it is, whether you could reasonably do a join against that column.
|
|
#
?
Feb 13, 2016 23:09
|
|
|
nielsm posted:Nah, "a list of things in one column" is a normal form violation. One way to think about it is, whether you could reasonably do a join against that column. I'm not sure which part of my post your "Nah" is referring to, but I get that a list inside a field violates 1NF. I'm just saying there's a difference between this: Products: (ProductId, Name, ExtraField1, ExtraField2, ExtraField3, ...) And this: Products: (ProductId, Name) ExtraFields: (ExtraFieldId, Name) ExtraProductFields: (ProductId, ExtraFieldId, Value) The idea is that you don't know at design time what's going to be in ExtraFields, and one of the things might turn out to be "ListOfSomething". If you're putting lists of things in Value then it's still a violation of 1NF, but it's a lot less crappy than the first version. I realize you could have an ExtraProductFieldParts with a foreign key to ExtraProductFields to allow lists without denormalization, but that would probably be less efficient if the most common case is for SomeArbitraryString to contain an atomic value, and it would complicate the design. Whether these "dynamic" tables are a good idea in the first place is a different matter (they're basically a small, non-typesafe database inside a typesafe database), but they seem to be pretty common in my experience.
|
|
#
?
Feb 14, 2016 00:13
|
|
|
nielsm posted:Nah, "a list of things in one column" is a normal form violation. One way to think about it is, whether you could reasonably do a join against that column. It is unacceptable. Specially if the "designer" uses a CHAR column to store a structure. Specially if that structure contains text, binary and packed (decimal) fields. Everything is nice and everyone is happy because in PL/I or COBOL you can define arbitrary overlays over memory blocks. But as soon as you try to downsize the application and read that information from a distributed system you see the JDBC driver trying to apply EBCDIC to UTF-8 translation to integers and packeds. So you have to cast the column to CHAR(x) FOR BIT DATA. And then you have to do the conversions yourself, including the big endian to little endian stuff. Lots of fun.
|
|
#
?
Feb 14, 2016 01:23
|
|
|
I think that problem is less of a database one (binary blobs are perfectly acceptable as far as I'm concerned) and more of a "serializing things by dumping the raw memory" problem. Which, at the very best, is rather dangerous assuming you're dealing with perfectly uniform hardware. Anything else and it'll blow up sooner or later. That goes for sending things over a network or writing them to a file or a bunch of other things that don't have much to do with databases. Edit: Oh, and using CHAR columns for storing binary data is obviously dumb, too.
|
|
#
?
Feb 14, 2016 01:58
|
|
|
For databases that support it, I like sweeping "expansion-friendly" data into JSON/XML data types where normalizing isn't really feasible. At work, somebody tried to do this many years ago, but used a varchar column instead of XML... now it's full of definitely-not-XML and there's a bunch of janky methods that people use to query against it. Migration is almost out of the question because it's literally the largest and most important (from a legal perspective) table.
|
|
#
?
Feb 14, 2016 02:28
|
|
|
Someone at my job decided the best way to log API requests was in a table with two columns, IDENTITY and a VARCHAR(MAX) that holds the JSON object of the request. These JSON objects are massive, because they log a lot of information related to telephony jitter and statistics. We literally cannot do anything with this table because to filter on it in any useful manner you need to parse the JSON in the VARCHAR column, and doing that makes the DB grind to a halt.
|
|
#
?
Feb 14, 2016 02:44
|
|
|
A co-worker a long ways back did a mysql disaster like that. Idiots making stupid schemas aren't uncommon, there's a million stories about it around the internet. What got me is when he started the task he assured the entire group he'd "done mysql before" and was familiar with how to do things so it would work well. It was a store for syslog messages so we could search for certain failures and alert on them. His grand idea was to store everything as strings: dates, messages, facilities.. all of it. After feeding 500+ nodes into this database and getting mad that his queries were taking too long, he slapped an index on every column and set up an even worse scheme of rotating tables once a week to keep them from getting too big. After he quit (but not before he got promoted into management) I deleted the whole stupid thing and replaced it with rsyslog.
|
|
#
?
Feb 14, 2016 02:56
|
|
|
Asymmetrikon posted:Someone at my job decided the best way to log API requests was in a table with two columns, IDENTITY and a VARCHAR(MAX) that holds the JSON object of the request. These JSON objects are massive, because they log a lot of information related to telephony jitter and statistics. We literally cannot do anything with this table because to filter on it in any useful manner you need to parse the JSON in the VARCHAR column, and doing that makes the DB grind to a halt. Scanning an entire table will tend to do bad things to DB performance, yeah. xzzy posted:It was a store for syslog messages so we could search for certain failures and alert on them. His grand idea was to store everything as strings: dates, messages, facilities.. all of it. After feeding 500+ nodes into this database and getting mad that his queries were taking too long, he slapped an index on every column and set up an even worse scheme of rotating tables once a week to keep them from getting too big. Indices make queries faster therefore more indices will make things go even faster. If things are still slow, then that means you need to start adding composite indices. All of them, ideally. This is where he failed. It's amazing the number of people who somehow believe something along those lines.
|
|
#
?
Feb 14, 2016 03:34
|
|
|
YeOldeButchere posted:Scanning an entire table will tend to do bad things to DB performance, yeah. Maybe I should bring up the genius at an old job who decided to implement an RDBMS on top of a NOSQL DB because NOSQL is better. Every query was an index, so he limited the maximum number of different queries for performance reasons.
|
|
#
?
Feb 14, 2016 03:42
|
|
|
Kazinsal posted:Here's some coding horror in the prosumer products world: http://appleinsider.com/articles/16/02/12/bug-in-adobe-creative-cloud-updater-erases-root-level-mac-data quoting this actually funny post in the vain hope that it will not get drowned out by people talking about database normalization
|
|
#
?
Feb 14, 2016 22:11
|
|
|
Kazinsal posted:Here's some coding horror in the prosumer products world: http://appleinsider.com/articles/16/02/12/bug-in-adobe-creative-cloud-updater-erases-root-level-mac-data But I wonder what Adobe was actually trying to do there. What is it supposed to be deleting?
|
|
#
?
Feb 14, 2016 22:22
|
|
|
The more things change, the more they stay the same: http://minimaxir.com/2013/06/working-as-intended/ (bungie's myth 2 installer did the same thing, except it would wipe out the entire hard drive)
|
|
#
?
Feb 14, 2016 22:35
|
|
|
nielsm posted:But I wonder what Adobe was actually trying to do there. What is it supposed to be deleting? Since it's in the context of an upgrade, I'm guessing the old "Adobe-whatever" installation folder? It's really sloppy to assume Adobe comes first, though.
|
|
#
?
Feb 15, 2016 01:16
|
|
|
LOOK I AM A TURTLE posted:This conversation inspired me to refresh my memory of 1NF, 2NF, 3NF, BCNF, etc. I've seen a lot of stupid stuff in database table design, but I don't think I've ever seen a table that fails to satisfy 1NF in any software I've worked on. The only thing that comes to mind is certain dynamic columns intended to contain arbitrary data that's validated elsewhere, possibly including comma-separated lists of values. But I think that's basically acceptable, if a little clumsy. I work at a non-technical company where our inventory database does not meet 1NF. We make a relatively small number of database accesses per day using a limited number of (never maintained) reports that are slow as a dog, despite checking only a few tens of thousands of records. It's not my job to improve this database in any way and I doubt any of my superiors would understand why or how it could be improved, so I leave it as is.
|
|
#
?
Feb 15, 2016 03:43
|
|
|
xzzy posted:The more things change, the more they stay the same: Steam at one point did as well when you ran it.
|
|
#
?
Feb 15, 2016 03:55
|
|
|
One of the EVE online installers would blow away boot.ini.
|
|
#
?
Feb 15, 2016 04:38
|
|
|
I have no hope that we will learn, only that we will stop letting software do this sort of thing. Regrettable: removing a method because you couldn't fix it. More regrettable: being unable to update to the version that removed said method because you use it everywhere.
|
|
#
?
Feb 15, 2016 05:34
|
|
|
Nude posted:Steam at one point did as well when you ran it. As far as anyone can tell, this was the FUSE NTFS driver crashing and then Steam reacting in the worst possible way to the "impossible".
|
|
#
?
Feb 15, 2016 07:56
|
|
|
Edison was a dick posted:Any of them? I think it rhymes with Hanker and W^&*er
|
|
#
?
Feb 15, 2016 08:21
|
|
|
YeOldeButchere posted:Scanning an entire table will tend to do bad things to DB performance, yeah. I worked on a project once, We had a database with 3 Tables, Data, Archive, Logs All 3 tables were Identical ID Primary Secondary Description Type Data (BLOB) We used the primary and secondary keys and description linked with type for the records, Any other field was stored in the Data blob as XML and we parsed Name Value pairs on loading each record into a HashMap Worked fine in development until we tried to use it to send 3 million SMS messages in a marketing push. - It then took 48 hours to send 300. We then ripped it apart and turned it into a decent Database design, - The original design was done by a contractor whos main job was senior developer at Credit Suisse
|
|
#
?
Feb 15, 2016 08:27
|
|
|
pseudorandom name posted:As far as anyone can tell, this was the FUSE NTFS driver crashing and then Steam reacting in the worst possible way to the "impossible". I thought it was because they were running rm -rf ${STEAMROOT}/* without bothering to check whether $STEAMROOT had been defined?
|
|
#
?
Feb 15, 2016 10:16
|
|
|
qntm posted:I thought it was because they were running rm -rf ${STEAMROOT}/* without bothering to check whether $STEAMROOT had been defined? Yup. FUSE NTFS was involved in discovering the bug, but the bug was the missing check for STEAMROOT.
|
|
#
?
Feb 15, 2016 11:34
|
|
|
TheresaJayne posted:
He probably came from an IMS-DB background. In IMS-DB is quite usual to define just the key fields and use a binary blob for the rest of the data. This way the developer has the  flexibilty of using the same segment (thats how records are called in IMS) to store different kinds of data. A lot of ex-IMS developers reconverted to the relational world do the same.Everything is fine until someone wants to do data mining over those tables. Then the ETL party begins.
|
|
#
?
Feb 15, 2016 13:19
|
|
|
There's a developer here who frequently makes fairly large commits of assorted unrelated changes with the title 'Committed pending changes'. 
|
|
#
?
Feb 15, 2016 14:23
|
|
|
I'm working on a project where some code is quite old. I was recently looking up some commits associated with some older parts of the codebase (I wanted to find the associated JIRA issues). All of the commits I found had 50-100 files in them, and each commit message referenced ~20 completely unrelated JIRA issues - the commit messages were just comma separated lists of issue keys. I assumed that the developer making these commits had been terrible, but when I asked a senior dev on my team about this, it turned out that this was standard practice. Apparently, for the first year of development, each single commit had to be approved by someone (who wasn't a developer - this wasn't code review), so in order to work efficiently, the commits were huge. I really hope I never have to go looking for old JIRA issues again.
|
|
#
?
Feb 15, 2016 15:08
|
|
|
chippy posted:There's a developer here who frequently makes fairly large commits of assorted unrelated changes with the title 'Committed pending changes'. I see "wip" as a commit message way more than I want to.
|
|
#
?
Feb 15, 2016 15:27
|
|
|
Illegal Move posted:in order to work efficiently, the commits were huge. Truly the mark of a great methodology.
|
|
#
?
Feb 15, 2016 16:16
|
|
|
Bognar posted:I see "wip" as a commit message way more than I want to. We have a developer who will often check things in with just 'phase 1 comp' and then there's 25 unrelated files across 3 different stories of work. And pretty much never a phase 2.
|
|
#
?
Feb 15, 2016 17:09
|
|
|
Nude posted:Steam at one point did as well when you ran it. The nVidia GeForce Experience crap rm -rf C:-ed my desktop a couple years ago when it updated itself. It was really confusing because everything I was already using kept working for the most part because files were locked, so it couldn't nuke everything and I have automated nightly backups, so I wasn't too hosed. Took a long time for me to stop making a manual backup before letting it do anything again.
|
|
#
?
Feb 15, 2016 17:44
|
|
|
qntm posted:I thought it was because they were running rm -rf ${STEAMROOT}/* without bothering to check whether $STEAMROOT had been defined? STEAMROOT is unconditionally set to the path of the directory containing the currently executing script. At some point between the script starting execution and the calculation of STEAMROOT, the directory containing the script vanished.
|
|
#
?
Feb 15, 2016 17:53
|
|
|
chippy posted:There's a developer here who frequently makes fairly large commits of assorted unrelated changes with the title 'Committed pending changes'. "Reverted accidental changes" is your next move. If she'd meant them to be part of the mainline, she'd have documented them.
|
|
#
?
Feb 15, 2016 17:57
|
|
|
Nude posted:Steam at one point did as well when you ran it. The deeper I go into the comments the spergier it gets quote:All bug fixes from Steam/Valve will be wrong unless they do a cultural change in how they write their software. edit: jesus christ, the last one lol quote:Solution: Just don’t delete user data. Ever. Slanderer fucked around with this message at 18:48 on Feb 15, 2016 |
|
#
?
Feb 15, 2016 18:41
|
|
|
Slanderer posted:The deeper I go into the comments the spergier it gets Daily-driver Linux users are spergy?! 
|
|
#
?
Feb 15, 2016 19:04
|
|
|
My professional life revolves around linux, and I'm not spergy.  Well, maybe a little. But I am self aware and can make jokes about it.
|
|
#
?
Feb 15, 2016 19:45
|
|
|
By "Daily Driver" I mean "uses it at home voluntarily because they really want to"
|
|
#
?
Feb 15, 2016 20:23
|
|
|
|
| # ? May 17, 2024 18:53 |
|
|
xzzy posted:I am self aware and can make jokes about it.
|
|
#
?
Feb 15, 2016 20:24
|
|
