Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Data engineering: what are you doing in my data swamp?!
 
  • Post
  • Reply
Hadlock
Nov 9, 2004

monochromagic posted:

Some thoughts -
3) I'm a bit curious - are you implementing custom CI/CD for each new project? This touches a bit on platform engineering but perhaps this could be standardised as well, e.g. prefer docker images or something similar that you then have a unified CI/CD process for. I'm trying to move us towards more docker and using the Kubernetes executor in Airflow for example - that's how we run dbt in prod.

I'm curious what is the dividing line between data engineering, and devops/platform engineering/sre

For example we had an "analytics" department that was maintaining their own spaghetti code of bash and SQL, database creation/modification, as well as a home built kitchen/pentaho/spoon ETL disaster and ftp/sftp stuff

I rewrote most everything but the sql and ported them to a new platform (with an eye towards moving them to airflow as the next step), parallelized a lot of the tasks and got the run time way down

On the other hand I've got a coworker that's building out some data pipelines for Google forms and excel data and piping it into tableau for upper management to do internal business forecasting. Arguably you could farm that out to a devops guy

* # ¿ Feb 26, 2024 21:10
  • Quote
Adbot
ADBOT LOVES YOU

# ¿ May 21, 2024 09:21
  • Quote
Hadlock
Nov 9, 2004

CompeAnansi posted:

It depends on the company. If you have a devops/platform team, then generally they do the job of maintaining the tools, and DEs just use them to accomplish data tasks. If you're small enough that you don't have a dedicated team, then DEs generally have to setup, maintain, and use the tools. If you're really small, they might expect an underpaid analyst to do all that and create charts too.

My experience has been that companies only allow specialists when they're forced to. So, if they've been forced to setup devops teams already, then the generalist DE doesn't need to do that bit of work, otherwise it's all on them. And so on.

I mean, is data engineering a sub-discipline of devops, like devsec ops or platform engineering, or is it somehow independent? Arguably it's a sub discipline of platform engineering

* # ¿ Feb 27, 2024 21:15
  • Quote
  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply
The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Data engineering: what are you doing in my data swamp?!