Data engineering: what are you doing in my data swamp?!

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Data engineering: what are you doing in my data swamp?!

Hughmoris: Apr 21, 2007; Let's go to the abyss!

For those interested in checking out the Microsoft flavor of Data Engineering:

You can create a free Microsoft 365 Developer Sandbox. This gives you free access to the Microsoft Fabric trial, which is Microsoft's next push for all things data. So you can spin up data lakes, warehouses, Spark clusters etc... No credit card needed for any of it, so you can play around without fear of racking up bills.

# ¿ Dec 29, 2023 19:43

Adbot: ADBOT LOVES YOU

# ¿ May 17, 2024 17:03

Hughmoris: Apr 21, 2007; Let's go to the abyss!

For those of you currently working as Data Engineers, what does your day-to-day look like? What technologies are you working with?

# ¿ Jan 3, 2024 00:46

Hughmoris: Apr 21, 2007; Let's go to the abyss!

Seventh Arrow posted:

Right now I'm working with the three technologies I hate the most: Salesforce, Informatica, and Oracle. However, the team is great so I don't mind. They're in the process of migrating Informatica to GCP, so I'm currently working on getting my Google cert. I've also let my python and SQL skills lag, so I need to pick that up again in case I have to do interviews when my contract is done.

I heard that working with Oracle was an invitation to Planet Money. Is that not the case?

Maybe it was just for Oracle DBA training.

# ¿ Jan 3, 2024 15:57

Hughmoris: Apr 21, 2007; Let's go to the abyss!

QuarkJets posted:

Every single person I know how has spent appreciable time working with oracle databases seems to seriously hate them, but I don't have any personal experience with Oracle DB so I don't really know why

ElehemEare posted:

I have administered Oracle 11g/12c on RAC, it was not a one-way ticket to Planet Money.

I've been sold a bill of lies. :mad:

CompeAnansi posted:

Our stack is:

...

I appreciate the insight. I really need to sit down and spend a little time exploring Airflow.

Do your data engineers also administer your Postgres instance, or do you have a dedicated DBA team for that?

# ¿ Jan 3, 2024 20:35

Hughmoris: Apr 21, 2007; Let's go to the abyss!

Seventh Arrow posted:

resume stuff

Looks fine to me but I'm no guru on the subject.

Has anyone here stood up a spark cluster? I've only used Databricks which provisions and handles everything behind the scenes but I'm curious how difficult it would be to stand up a 3-node cluster on EC2s? Just as a learning exercise.

# ¿ Feb 21, 2024 18:45

Adbot: ADBOT LOVES YOU

# ¿ May 17, 2024 17:03

Hughmoris: Apr 21, 2007; Let's go to the abyss!

BitAstronaut posted:

For context, I'm currently a junior data engineer working in finance - graduated two years ago. My day-to-day is usually something along the lines of

- Company wide standup (Mondays only)
- Go over any code reviews I have and make notes ahead of my morning standup
- Morning standup with my team where we go over what we accomplished, what we are doing for the day, and any blockers we may have
- I'll join the debug session immediately following the standup if I have anything to go over.
- Work on my current tasks.

We also have an on-call rotation for seven day periods where your days are usually restarting Airflow DAGs, dealing with data issues, and answering questions from the business.

We use Airflow for orchestration, dbt, Postgres, K8s, and a ton of in-house tooling using Python and Java.

Thanks for the insight!

# ¿ Mar 7, 2024 23:39

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Data engineering: what are you doing in my data swamp?!