|
YanniRotten posted:I've had PRs land on my desk from a contractor where they were doing at least some of their filtering in memory after pulling in all rows, and were doing their own aggregations (literally they did not preserve ANY individual row data for what they were doing). It might be them not knowing that SQL can do that. On the other hand, if the database is storing all the intermediate results on disk, doing stuff in-memory can be much faster. I've seen a query go from three hours to eight minutes when we stopped using SQL for joins and aggregations.
|
#
?
Aug 28, 2020 19:02
|
|
|
|
| # ? May 9, 2024 14:44 |
|
|
YanniRotten posted:I've had PRs land on my desk from a contractor where they were doing at least some of their filtering in memory after pulling in all rows, and were doing their own aggregations (literally they did not preserve ANY individual row data for what they were doing). I had to fix a recent production issue that was like this, with the added bonus of it being DynamoDB results. The dev didn't know results were paginated so once the results were >1MB data just disappeared...
|
|
#
?
Aug 28, 2020 19:23
|
|
|
JehovahsWetness posted:I had to fix a recent production issue that was like this, with the added bonus of it being DynamoDB results. The dev didn't know results were paginated so once the results were >1MB data just disappeared... Don't the SDKs ship pagination-aware wrappers anyway? This is the double whammy of using an API that's at a lower level than you really need but also not reading the docs for it.
|
|
#
?
Aug 28, 2020 19:37
|
|
|
Hollow Talk posted:When you need to calculate things, just do SELECT * FROM, pull everything into python, then insert it all again after your calculation is done by iterating over all rows and inserting each one individually. Do this in pandas if you are a Data Scientist. Put SQL 5/5 on your resume. At my very first programming job I was treated like a wizard for massively increasing the performance of this desktop app they were using. Every single database call involved pulling down the entirety of whatever table you needed. Need to get one order? Pull down the entire orders table and find order 12345. Don't forget to pull down the entire address, customer, phone number, and notes tables and pull out all of the ones you need too! Just going through and removing a bunch of .ToList() and for loops and replacing them with .Where fixed so much.
|
|
#
?
Aug 28, 2020 19:47
|
|
|
wilderthanmild posted:At my very first programming job I was treated like a wizard for massively increasing the performance of this desktop app they were using. Every single database call involved pulling down the entirety of whatever table you needed. Need to get one order? Pull down the entire orders table and find order 12345. Don't forget to pull down the entire address, customer, phone number, and notes tables and pull out all of the ones you need too! Just going through and removing a bunch of .ToList() and for loops and replacing them with .Where fixed so much. Those jobs where you’re a genius from just sort of being able to do things right are fun. I got an insane amount of mileage at my first job just out of having any awareness of indexes. Oh this terrible process takes an hour how about five seconds instead with me being 100% sure nothing has been broken by the change. I mean, you gotta get out of those places fast, smartest person wrong room etc., but it’s kind of fun building that reputation out of stuff everyone should know how to do at a good shop.
|
|
#
?
Aug 28, 2020 20:32
|
|
|
Our team maintains our own ORM and I am completely sold that it might be ideal to just build it yourself. The advantages are that you get to build it exactly how you want, adding features is very easy, and you are expected to have at least some understanding of SQL as a team which conveys some nice benefits.
|
|
#
?
Aug 28, 2020 21:10
|
|
|
Yeah nowadays I just use Dapper and do most of my code in either stored procedures that abstract db structure away OR bring data that we will need into memory and cache whatever makes sense (often times enums or lookup tables). Performance is a lot better and I don’t have to wrestle with EF to keep it from doing something extremely stupid.
|
|
#
?
Aug 28, 2020 23:44
|
|
|
Canine Blues Arooo posted:Our team maintains our own ORM and I am completely sold that it might be ideal to just build it yourself. The advantages are that you get to build it exactly how you want, adding features is very easy, and you are expected to have at least some understanding of SQL as a team which conveys some nice benefits.  Knowing SQL when dealing with a RDBMS is essential. Building your own ORM when there are gazillions to choose from is just  .
|
|
#
?
Aug 29, 2020 01:00
|
|
|
Volguus posted:
Yeah... if the benefit is that you know exactly what it does, that's going to erode over time unless your documentation is up to the standard of existing ORMs intended for wide consumption. Anyone you onboard is gonna be like "WTF is this". I'm against rolling your own bespoke stuff unless your use case is super special (it usually isn't). There's options out there and the maintainers have thought through a LOT of stuff and have good documentation.
|
|
#
?
Aug 29, 2020 01:10
|
|
|
We rolled our own ORM but we have an extreme benefit of a relatively small surface area of what is involved. I can see using an off the shelf solution if your core DB schema is constantly expanding to support new features, but our features primarily evolve by way of using existing ORM pieces as lego bricks essentially. Off the shelf ORMs just add a lot of overhead that *may* save you time and headache, but it also may *add* time and headache too. Depends on the situation.
|
|
#
?
Aug 29, 2020 02:33
|
|
|
ultrafilter posted:It might be them not knowing that SQL can do that. On the other hand, if the database is storing all the intermediate results on disk, doing stuff in-memory can be much faster. I've seen a query go from three hours to eight minutes when we stopped using SQL for joins and aggregations. mysql?
|
|
#
?
Aug 29, 2020 03:27
|
|
|
shoeberto posted:We rolled our own ORM but we have an extreme benefit of a relatively small surface area of what is involved. I can see using an off the shelf solution if your core DB schema is constantly expanding to support new features, but our features primarily evolve by way of using existing ORM pieces as lego bricks essentially. This was kind of our problem space and our concerns. Off the shelf stuff *can* make things easy, but rarely does exactly what you want, and depending on how 'exactly' you need the tooling and how difficult it'd be to build for your project (I think our's took a week of one person's time), it might make sense to just build it. I totally get the idea that it'd be a bad idea for some projects and teams, but even in hindsight, there is no way I would just do the same thing. We get so much mileage out of our solution and the cost was quite low.
|
|
#
?
Aug 29, 2020 04:46
|
|
|
One time I used a handmade ORM that I really liked. It was built up over ten years to work really well with an e-commerce site. It only did 80% of the job that a proper, fully-featured ORM would do, but since it’s was built for their use, it was the right 80%.
|
|
#
?
Aug 29, 2020 05:21
|
|
|
When I first started at Experts Exchange, they were using a home-built ORM they wrote in Perl. Nobody really knew Perl, so nobody could ever fix anything or make improvements. I rewrote the thing in Java, which everybody knew, and even introduced the concept of having the generated code call utility functions, so we could make bug fixes in those, instead of tweaking the code generator, regenerating everything, and seeing what happened. I didn't write any tests, though, because we weren't doing that there, yet, lol. They're still using that dumb ORM. And the horrible, home-built CMS that originally defined content using XML that was stored in the database, with everything having a revision number attached to it. I rewrote that, too, to just store the XML as files that could then be checked into normal version control. The jerk who wrote that terrible CMS is in our group chat, along with the rest of us former employees, and I always have to bite my tongue about how it was my mission for a while to get rid of every line of code with their name on it.
|
|
#
?
Aug 29, 2020 05:36
|
|
|
Writing one or more terrible CMSes is a rite of passage for developers. I have like 4 under my belt.
|
|
#
?
Aug 29, 2020 14:07
|
|
|
I just left a terrible, terrible job, and god I'm glad to be out of there. The tech stack was boring, you were locked into a very rigid architecture where a few good decisions were outweighed by the massive tech debt and lack of flexibility and outright horrible decisions. The management was so awful, they'd promise our clients things without consulting with engineering, set arbitrary deadlines for them, and yell at us when they inevitably weren't able to be delivered. Asked to work late a lot, and without a real team lead all that responsibility fell on my shoulders. I demanded more pay and a title change if I was going to do the extra work, and my boss pushed for it (I saw the emails, he showed me them on my last day), and they just...refused? Because I refused to work late constantly? And then after they refused to give me the promotion they still kept demanding I work late? This ain't a charity. In my last week, they assigned me a big, mission critical feature to finish by Friday. This was because they didn't trust anyone else on the team to finish it and when I asked who was going to maintain it when I was gone, they told me to not worry. I mean, it's my last week and I'm not gonna break my back over this, but this felt like a really bad decision for the business. Whatever, I ended up spending my last week trying to answer any and all questions my team had and fighting fires (which are just constant). I pushed up a work in progress branch with the feature on Thursday afternoon and I got it working 95%, so I figured I'd spend the last Friday morning fixing that last 5%. Well, at 8:30PM on Thursday, I get a phone call from someone asking me to join a call to explain what I was doing so they could finish it overnight. I mean I refused, because it's my last day. But more worrying for the future of the company was that my teammate, who I walked through the technical decisions on this, was explaining it to these supposedly very seasoned engineers at the company and they didn't understand it? Like, nothing was complicated? It's literally just building on a feature that's pretty tried and true. So they just....ripped out all my code and rewrote some garbage hacks that I knew would only half work. I was right because I spent the Friday morning working with the dude who was tasked to rewrite it (he had no context whatsoever on the project so ... good luck guys) and we fixed a few bugs but I also saw it didn't really work (but it looked like it did at first glance). Well, not my problem, good job idiots. Also how are you spending my last day just bombarding me with questions and asking me to fix things, it's my last day, maybe you should've planned my departure better? Good loving riddance. They also cut off my access in the middle of a final call with my boss, which is wild.
|
|
#
?
Aug 29, 2020 15:05
|
|
|
good on you for getting outta there, let us know when they continue asking questions while you are no longer employed there
|
|
#
?
Aug 29, 2020 18:06
|
|
|
Don't forget when they realize they're not about to do what they're asked to demand boat money to help again
|
|
#
?
Aug 29, 2020 18:56
|
|
|
Volmarias posted:Don't forget when they realize they're not about to do what they're asked to demand boat money to help again $300/hr 80 hours minimum, double rate for anything before 10AM or after 5PM. 'Let' yourself be talked down to 'only' $200.
|
|
#
?
Aug 29, 2020 19:33
|
|
|
Telling a former employer you'll be happy to work as a paid consultant is the fastest way to learn just how important you were to the organization Hint: you weren't at all
|
|
#
?
Aug 29, 2020 20:43
|
|
|
Blinkz0rz posted:Telling a former employer you'll be happy to work as a paid consultant is the fastest way to learn just how important you were to the organization Look a forum poster had this happen to them, we can all dream ok?
|
|
#
?
Aug 29, 2020 22:07
|
|
|
Not sure I'd actually take any payment to work for them again, in any capacity, to be honest. I took a (slight) pay cut to leave and the counteroffer was considerable.
|
|
#
?
Aug 29, 2020 23:18
|
|
|
I worked briefly in a shop like that, and it's just not worth the drain on your mental health when there are good shops out there that will actually treat you with dignity.
|
|
#
?
Aug 29, 2020 23:35
|
|
|
Volmarias posted:Look a forum poster had this happen to them, we can all dream ok? I did some work for a former employer as a paid consultant but we parted on good terms.
|
|
#
?
Aug 30, 2020 03:23
|
|
|
prom candy posted:I did some work for a former employer as a paid consultant but we parted on good terms. I'm currently working for my previous employer. I quit on good terms as a full time employee, then returned later as a private contractor for just over twice my previous rate. It was originally a short, fixed length contract to finish off a specific project. A year and six contract extensions later, I'm still there working on things which I guess fall into a specific cross-section of knowledge/niche where they struggle to find a replacement. They are currently going through a restructering, the changes are at every level and significant so I imagine there won't be any further contract extensions.
|
|
#
?
Aug 30, 2020 04:11
|
|
|
Blinkz0rz posted:Telling a former employer you'll be happy to work as a paid consultant is the fastest way to learn just how important you were to the organization Sorry, can't hear you over the money rolling in 
|
|
#
?
Aug 30, 2020 06:51
|
|
|
What ORMs, specifically, are giving y'all trouble?
|
|
#
?
Aug 30, 2020 12:15
|
|
|
Ither posted:What ORMs, specifically, are giving y'all trouble? All ORMs that generate SQL eventually encounter the same problems: - They generate awful performance-killing SQL - They generate incorrect SQL - Trying to do something simple requires a huge investment of time and energy, and often results in case #1 or #2. I've seen this with both nHibernate and every version of Entity Framework released in the past decade. I'm sure other platforms have the same issues as well.
|
|
#
?
Aug 30, 2020 16:19
|
|
|
The only parts of ORMs that are worthwhile are mapping tools (just to copy rows of data into your own objects and vice versa) and perhaps type safe query builders. Any kind of lazy loading, session tracking, auto-generated SQL etc is all garbage
|
|
#
?
Aug 30, 2020 17:09
|
|
|
ActiveRecord is pretty good for basic use case stuff, it's really annoying when you need to go beyond that though.
|
|
#
?
Aug 30, 2020 17:45
|
|
|
I feel like the problem with ORMs is that they're "fine" until they aren't, and then it's a headache and a half to debug. Standard SQL database with stored procs is pretty easy to debug. At least, that has been my experience.
|
|
#
?
Aug 30, 2020 19:36
|
|
|
Objects and relations are different, mathematically speaking. In order to get the full advantage out of an ORM, you have to forego some of the advantages of one or the other.
|
|
#
?
Aug 30, 2020 19:39
|
|
|
These all sound like arguments to either use the ORM for some things and write SQL for the more complex stuff or to only use SQL. I've not seen a good reason to roll your own ORM
|
|
#
?
Aug 30, 2020 19:52
|
|
|
Part of the fun of rolling your own replacement for readily available libraries is inventing a justification after the fact
|
|
#
?
Aug 30, 2020 20:03
|
|
|
Protocol7 posted:I feel like the problem with ORMs is that they're "fine" until they aren't, and then it's a headache and a half to debug. I'm guessing you've never had to work with the output of a dev who realized "if I try to check my business logic into application code, people whine about 'documentation' and 'meaningful variable names' and all that useless garbage - but if I write a sproc then I can do whatever I want!"
|
|
#
?
Aug 30, 2020 20:05
|
|
|
One of the major benefits of leaving a job is not having to think about any of those problems again... taking a consulting job kinda defeats the point.
|
|
#
?
Aug 30, 2020 21:50
|
|
|
Space Gopher posted:I'm guessing you've never had to work with the output of a dev who realized "if I try to check my business logic into application code, people whine about 'documentation' and 'meaningful variable names' and all that useless garbage - but if I write a sproc then I can do whatever I want!" That sounds like an egregiously bad developer even by our jaded standards. Doubt problems from their code are only endemic to SQL.
|
|
#
?
Aug 31, 2020 14:04
|
|
|
Slimy Hog posted:These all sound like arguments to either use the ORM for some things and write SQL for the more complex stuff This is basically it, in my experience. Couple projects ago had a home-rolled ORM-but-not because the client had a "no hibernate" stipulation (it was more complicated than that) that was effectively just a "here's some stuff for basic crud operations and some basic filtering, if you want anything interesting you've gotta write your own SQL" and it honestly worked out pretty well as far as SQL horrors go.
|
|
#
?
Aug 31, 2020 18:06
|
|
|
Are people using "ORM" to also include "database access layer" cause in my experience something that just marshalls data in and out of relatively flat objects for CRUD operations is just DAL and not ORM.
|
|
#
?
Aug 31, 2020 19:31
|
|
|
|
| # ? May 9, 2024 14:44 |
|
|
Paolomania posted:Are people using "ORM" to also include "database access layer" cause in my experience something that just marshalls data in and out of relatively flat objects for CRUD operations is just DAL and not ORM. I probably am, given that in retrospect our "ORM" had no relationship modelling 
|
|
#
?
Aug 31, 2020 19:43
|
|