Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Methanar
Sep 26, 2013

by the sex ghost

Hadlock posted:

"bro" recruiter who didn't have any understanding of what we did

Fascinating.

The guy I interviewed today was from Adobe and i liked him. More than anything he made Adobe actually sound pretty awesome as a place to work.

Adbot
ADBOT LOVES YOU

Hadlock
Nov 9, 2004

I'll reiterate that I'm hesitant to even comment on "bro" culture but he did in fact have a popped polo collar and would sort of hop back and forth from one foot to the other with longish messy frat boy hair and hat in an impersonation of every frat guy in every college movie ever that almost had him begging people to ask him if he was a parody of someone, also called me "bro" constantly

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Hadlock posted:

Ask them what percentage of their time will be spent on IC work, if that number is higher than 0 should be an immediate pass

I manage a team of developers and I will occasionally pick off a ticket of my own to work on. Usually something that is small and not time sensitive, since my schedule is interrupt driven. I like that it forces me to keep my development environment up to date, which I can use to help my team get unstuck when they run into issues.

22 Eargesplitten
Oct 10, 2010



I'm doing cloud engineer stuff but most of it is manual (I've gotten management buy-in to use terraform but half the administrators refuse to use it which kind of defeats the purpose on anything I don't solely own). I want to get into DevOps/SRE roles, it seems like a good starting point would be getting some Terraform projects on my github and getting the CKA since K8s is locked up tight at my current org. I'm midway through a Terraform project but taking a break to study for the CKA. Is that something I could feasibly do without doing the whole Linux Foundation pathway? I work in Linux on a daily basis although I'd say I'm still a junior administrator, I can get around in the bash CLI. I have done a dockerized app as an interview test and run a pre-packaged Octoprint package. I have a Pluralsight subscription and am partway through the first course on their CKA pathway. I am having a hard time absorbing anything but I feel like that's because this guy frontloads how everything works before we even install anything.

Are there any things I should absolutely make sure I know before I bite the bullet and sign up for the exam? I feel like setting a deadline for myself by putting my $375 down is going to motivate me. I was thinking maybe midway through June, a month and a half. Is that reasonable to go from 0 to 60 on K8s given that I have no social life or family?

Walked
Apr 14, 2003

22 Eargesplitten posted:

Is that reasonable to go from 0 to 60 on K8s

No

22 Eargesplitten posted:

given that I have no social life or family?

…maybe


Protip: O’Reilly Safari (or whatever they rebranded it to) has EXCEPTIONAL live training sessions (instructor led, live) which are way above the price point of the service (plus books - which is why I joined way back when).

They have several CKA courses that run periodically and are genuinely solid and worthwhile. Plural sight is great but nothing beats asking questions to an expert, live

Methanar
Sep 26, 2013

by the sex ghost
k8s is easy. It's building and integrating it into the rest of the ecosystem of your org such that it actually benefits the business that's hard.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Why CKA over CKAD? CKA is focused on cluster management (I.e. The stuff that every managed kubernetes service handles for you) and has limited value if you're not managing your own cluster nodes. CKAD is focused on actually operating applications within the cluster, which has a lot more utility.

https://github.com/dgkanatsios/CKAD-exercises

I used that to prep and it took me a few days to get everything down. I barely passed the exam though, so I may have just gotten lucky.

22 Eargesplitten
Oct 10, 2010



I had a recruiter recommend it as something that a lot of employers were looking for but maybe he was wrong. Basically whatever the equivalent of the AWS Certified Solutions Architect Associate is, I want that.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

22 Eargesplitten posted:

I had a recruiter recommend it as something that a lot of employers were looking for but maybe he was wrong. Basically whatever the equivalent of the AWS Certified Solutions Architect Associate is, I want that.

I have no idea what that aws cert is but there are two main Kubernetes certifications:

Administrator
Application Developer

Administrator focuses on standing up a cluster and managing the nodes. Something you won't be doing at all if you're using a cloud based managed Kubernetes service like EKS/AKS.

Application Developer focuses on actually running containerized workloads within a cluster -- designing deployment manifests, various types of networking options, troubleshooting application failures, doing rolling deployments, etc.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Methanar posted:

k8s is easy. It's building and integrating it into the rest of the ecosystem of your org such that it actually benefits the business that's hard.
This is where the utility and the value proposition both get a little bit hairy, because integration into the ecosystem is the entire reason you'd want to use Kubernetes over a batteries included but opinionated towards a bunch of poo poo integrations no one wants solution like ECS/Fargate in the first place.

But this is a great callout, because this is one circumstance where I see deployments like Kubernetes fall down over and over: when the business doesn't actually have an ecosystem, it has a jumbled mess of components handed down from a decade or more of disorganized technical decision-making. The company thinks choosing a big standard platform as a core pillar of its technical strategy will drive simplification of the internal ecosystem, but it doesn't and never will. What it does provide is a force multiplier for the teams spearheading those initiatives, who now have a common language and approach with loose couplings at predefined and usually sensible boundaries.

In other words: we're at a point on the adoption curve where if you haven't started with managed Kubernetes in greenfield, by the time you're big enough to have an entrenched problem that's now worth solving with Kubernetes, you're probably dropping it into the middle of an organization with big problems.

The Iron Rose
May 12, 2012

:minnie: Cat Army :minnie:
Will taking one of those certs explain to me why I still need to write pre-stop sleep commands for 500-less no downtime webapp deployments?

Is that still the recommended advise? I recall several articles and one kubecon presentation about the matter and remain irked that it’s necessary.

Context: https://www.youtube.com/watch?v=0o5C12kzEDI

also

https://blog.gruntwork.io/delaying-shutdown-to-wait-for-pod-deletion-propagation-445f779a8304


Like I can read and watch these and get why it’s needed. But still! Feels bad man

Methanar
Sep 26, 2013

by the sex ghost

Vulture Culture posted:

The company thinks choosing a big standard platform as a core pillar of its technical strategy will drive simplification of the internal ecosystem, but it doesn't and never will. What it does provide is a force multiplier for the teams spearheading those initiatives, who now have a common language and approach with loose couplings at predefined and usually sensible boundaries.
This is a little bit semantical. For the teams choosing to spearhead the 'fix this poo poo' initiatives, the use of a big standard platform certainly in my mind does provide simplification. Yeah I mean a hammer doesn't hammer nails on its own, the guy swinging the hammer does. But maybe it's just my naivety to think that everyone else implicitly assumes and reads 'tool is good, when properly used'

Vulture Culture posted:

In other words: we're at a point on the adoption curve where if you haven't started with managed Kubernetes in greenfield, by the time you're big enough to have an entrenched problem that's now worth solving with Kubernetes, you're probably dropping it into the middle of an organization with big problems.

Yeah that's why I have a job.

Methanar fucked around with this message at 05:33 on Apr 25, 2022

LochNessMonster
Feb 3, 2005

I need about three fitty


22 Eargesplitten posted:

Is that something I could feasibly do without doing the whole Linux Foundation pathway?

I’d rec the CKAD exam over CKA as well, but wouldn’t take the linux foundations courses on it. It’s the worst online learning platform I’ve ever used. Instructions are ok, platform itself is not that good and the exercises are bad. It also has little to no support for questions.

Buy a course on ACG, Udemy and take that instead.

22 Eargesplitten
Oct 10, 2010



I'll take the Pluralsight course first and see where I get from there. It seems like this course is from around k8s 1.18, is there anything huge that has changed since then aside from no longer calling it a master node and Docker being pushed aside?

This is kind of my first venture into the "dev" side of devops so here's hoping it goes well.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

22 Eargesplitten posted:


This is kind of my first venture into the "dev" side of devops so here's hoping it goes well.

Good news: this isn't the dev side of devops. Putting an app inside a container and running it in a kubernetes cluster is still firmly an operational activity.

22 Eargesplitten
Oct 10, 2010



Oh, it’s called “for developers” so I thought it would be development focused.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

22 Eargesplitten posted:

Oh, it’s called “for developers” so I thought it would be development focused.

It's just how they distinguish "managing the cluster" from "managing the stuff running in the cluster". You don't need to program to do either.

If I were naming the certs, they'd be certified kubernetes cluster administrator and CK Workload Operator. Those are better descriptors of what the content actually covers.

Doc Hawkins
Jun 15, 2010

Dashing? But I'm not even moving!


the people who develop the services should also know how to operate them imo

we could call that something... developed-operations? i'll workshop it

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Doc Hawkins posted:

the people who develop the services should also know how to operate them imo

we could call that something... developed-operations? i'll workshop it

You'll get no disagreement from me on that. Unfortunately, in the real world, it often ends up with the same old pattern of "throw it over the wall", except now the operations team is the devops team and they have better source control and automation discipline than the operations team used to have. It's marginally better and completely misses the point of devops!

Hadlock
Nov 9, 2004

Is there an established plan of action for running Django migrations on logical postgres replicas

Migrations on physical/classic replication is a snap but there's a ton of moving parts to account for with logical

I think the answer is "no, don't use local replication that way" but curious if anything has changed recently

Hadlock fucked around with this message at 19:49 on Apr 26, 2022

deedee megadoodoo
Sep 28, 2000
Two roads diverged in a wood, and I, I took the one to Flavortown, and that has made all the difference.


How many of you are the cloud expenditure police in your organizations? Do you manage costs, budgets, reporting, purchasing reservations, or ownership of alerting teams when they are spending too much money? If so, how are you doing this? Do you use costexplorer or custom reports or some other tool? I'm trying to get an understanding of what other organizations are doing because my former director had written a lot of automated reports and alerts for the financial side of things. This includes reports that break down costs by team, sending daily reports to leads, and alerting if the daily spend exceeded certain thresholds. In my mind, these are not DevOps tasks and I don't want to own them and I've never had to own them in previous roles so I'm interested in casting a wider net and seeing what other orgs are doing.

Hadlock
Nov 9, 2004

Generally in my experience cloud costs are Not A Problem until there's a budgetary crisis somewhere else and then finance comes after you with a big stick to scapegoat. If your cloud costs are not outrageous and not climbing then having the reports is a good shield to tell finance to gently caress off and go find another scapegoat, because your house is in order. If you own those reports now, the best thing you can do is schedule a quarterly meeting with the analyst assigned to your group and use them to beat them into submission over the fact that your house is in order.

IMO these are absolutely devops tasks, your devops manager needs to prove to the rest of the company that you are competent in managing your own wheelhouse, and those reports are the language upper management speaks in

Hadlock fucked around with this message at 20:52 on Apr 28, 2022

Methanar
Sep 26, 2013

by the sex ghost

deedee megadoodoo posted:

How many of you are the cloud expenditure police in your organizations? Do you manage costs, budgets, reporting, purchasing reservations, or ownership of alerting teams when they are spending too much money? If so, how are you doing this? Do you use costexplorer or custom reports or some other tool? I'm trying to get an understanding of what other organizations are doing because my former director had written a lot of automated reports and alerts for the financial side of things. This includes reports that break down costs by team, sending daily reports to leads, and alerting if the daily spend exceeded certain thresholds. In my mind, these are not DevOps tasks and I don't want to own them and I've never had to own them in previous roles so I'm interested in casting a wider net and seeing what other orgs are doing.

FinOps is its own discipline. But in my org there is a collaboration between the k8s team and SRE to drive cost reductions by identifying cases where autoscaling makes sense, adopting spot instances, requests/limits are obviously just set badly, where things are overprovisioned, or it makes sense to migrate workloads off of AWS to DCs. Both SRE and the k8s team have invested effort in providing tools to help with the autoscaling concerns, including developing internal libraries for adaptive concurrency. Basically, naively scaling up one application to handle a huge spike in load may only kick the can down the road to where downstream services suddenly die under the deluge, adaptive concurrency can help provide some backpressure or slow start logic within the application. Not only the downstream microservices can die under the upstream scale up load, but other components as well including Kafka, postgres, redis, other APIs further down the chain, whatever. So in the sense that driving autoscaling and rightsizing projects is policing cost, then yes.

We do have a collaboration with the team that managed hardware procurement for the DCs and AWS reservations, since k8s is a very large consumer of those to make sure we forecast appropriately.
We also built a nice 'wasted compute resources' graph on the front page of an internal web tool so everyone gets visibility on where the money is going.

We use cloudhealth, but it sucks.

Methanar fucked around with this message at 20:58 on Apr 28, 2022

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender
Cost Explorer and budget alerts are fine for reporting & alerting of relatively small scale stuff. If you have many accounts, or if people need fine-grained data, you might need something more sophisticated. If you're multi-cloud, there's tools like Appt.io's Cloudability that can slurp out data from various clouds, normalize them, and present nice graphs and such. But if you're tasked with extracting and presenting the data, you may be at the behest of a finance group that has very specific data needs (and likely uses ancient tech to process it).

FinOps may not sound like DevOps, but there's a lot of overlap since usage + cost are so tightly related. And some cost savings initiatives require engineering work to achieve (e.g. designing workloads to run smoothly on spot instances). The overlap is large enough that the DevOps/FinOps teams shouldn't be distinct groups, imho.

I would not recommend leaving the users to decide what's best for themselves. Cost optimization is a really complex topic, and even if users realize their costs are high, they won't necessarily know the best way to get those costs down. A dedicated experienced FinOps group can notice problems (ideally before they become problems) and guide recommendations. Very broadly speaking, they can help navigate:
- Vendor relationship optimizations (savings plans, reserved instances, negotiate volume discounts, support contracts, etc)
- Admin optimizations (one-off settings & maintenance to reduce costs, e.g. security policies to keep out coin miners, locking down expensive instance types & regions)
- Engineering optimizations (e.g. modifying workloads to run smoothly on spot instances, and reducing cross-region traffic)

New Zealand can eat me
Aug 29, 2008

:matters:


Plorkyeran posted:

I would not recommend doing that unless you have very modest requirements. Github actions is not very good at keeping their macOS builders up to date (e.g. they still haven't updated to macOS 12 and so can't run Xcode 13.3) and even on the Enterprise plan it took us ~6 months to successfully get bumped up 50 concurrent macOS jobs.

Seconding this. They also had a breathtaking issue I called "Xcode roulette", where build time version checks would pass but it would build against a different version of swift and or the SDK. Made for some really nonsensical and hard to manually reproduce bugs for a few months, compounded by the random chance that the box you got when you archived your builds to send them off to the app store was also hosed. So if you were shipping updates often, these things would come and go with no explanation

I'm just now settling into a new devops job after a decade+ of iOS. Really impressed with self hosted GitLab so far. Their pipelines easily do all the things I wish I had when I was smashing Jenkins into itself. I'm just doing real straightforward poo poo so far like digging all of the secrets out of the codebases and shoving them in vault, and hooking various APIs together in webapps. It's weird and refreshing to get paid to do this kind of stuff, at A Real Company That Is Profitable instead of building another app for a startup that won't exist in 2 years. I hadn't realized how much of that pressure I had internalized until it wasn't there anymore!

Radia
Jul 14, 2021

And someday, together.. We'll shine.

Doc Hawkins posted:

the people who develop the services should also know how to operate them imo

we could call that something... developed-operations? i'll workshop it

the amount of times i've talked about docker containers and poo poo with engineers and they're like "this sounds like a system engineer's job?? i just write code" legitimately gives me aneurysms

New Zealand can eat me posted:

Seconding this. They also had a breathtaking issue I called "Xcode roulette", where build time version checks would pass but it would build against a different version of swift and or the SDK. Made for some really nonsensical and hard to manually reproduce bugs for a few months, compounded by the random chance that the box you got when you archived your builds to send them off to the app store was also hosed. So if you were shipping updates often, these things would come and go with no explanation

I'm just now settling into a new devops job after a decade+ of iOS. Really impressed with self hosted GitLab so far. Their pipelines easily do all the things I wish I had when I was smashing Jenkins into itself. I'm just doing real straightforward poo poo so far like digging all of the secrets out of the codebases and shoving them in vault, and hooking various APIs together in webapps. It's weird and refreshing to get paid to do this kind of stuff, at A Real Company That Is Profitable instead of building another app for a startup that won't exist in 2 years. I hadn't realized how much of that pressure I had internalized until it wasn't there anymore!

have you ever worked with circleci? i've been considering gitlab but honestly circleci does EVERYTHING i want so loving well so far

New Zealand can eat me
Aug 29, 2008

:matters:


Lady Radia posted:

the amount of times i've talked about docker containers and poo poo with engineers and they're like "this sounds like a system engineer's job?? i just write code" legitimately gives me aneurysms

I think part of why I ended up landing this role was because I made it pretty clear that I was the 'plumber' any time we needed one and don't mind chasing down dependency errors or whatever weird bs. Have always thought it was amusing how many devs shy away from owning this side of things. Some of the best career advice I ever got was from a rare cool manager: find the poo poo nobody else wants to do and loving own it, you'll be indispensable.

Lady Radia posted:

have you ever worked with circleci? i've been considering gitlab but honestly circleci does EVERYTHING i want so loving well so far

Nope, but I've also never heard of any traumatic horror stories caused by them which is always a plus. If you don't need to self host and it suits all of your needs, let it rip!

Super-NintendoUser
Jan 16, 2004

COWABUNGERDER COMPADRES
Soiled Meat
edit moving this to the Working in IT thread, it's more appropriate there:

https://forums.somethingawful.com/showthread.php?threadid=3653857&pagenumber=2199#post523114234

Super-NintendoUser fucked around with this message at 12:09 on Apr 29, 2022

asap-salafi
May 5, 2012

THUNDERDOME LOSER 2019
Become the hero and leave when people start taking the piss

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender

New Zealand can eat me posted:

Some of the best career advice I ever got was from a rare cool manager: find the poo poo nobody else wants to do and loving own it, you'll be indispensable.

... is that good career advice? Being indispensable means managers might prevent promoting you or letting you work on anything else (or god forbid restricting vacation) because... you're indispensable.

When you're just starting out in your career it might make sense to specialize in niche areas to differentiate yourself. But the only advantage I see to being indispensable or knowledgable about a niche is that you can negotiate higher comp.

New Zealand can eat me
Aug 29, 2008

:matters:


A key part of this analogy is that plumbers are not required to be there 100% of the time to make sure the poo poo is moving. Be a plumber not a cog

Super-NintendoUser
Jan 16, 2004

COWABUNGERDER COMPADRES
Soiled Meat

New Zealand can eat me posted:

A key part of this analogy is that plumbers are not required to be there 100% of the time to make sure the poo poo is moving. Be a plumber not a cog

If you are the only plumber that knows how to fix a widget you will be chained to that widget forever. Enjoy 1.5% raises a year while others sail on by in their careers.

Hadlock
Nov 9, 2004

Depends on what else you're doing with your life at the time

If you already have the house/the car/the boat/the wife/the kids, that might be a pretty good gig. Be home at 4pm every Friday, up at 6am every Saturday going skiing, or whatever, might want to just punch that time card for a couple of years

Love Stole the Day
Nov 4, 2012
Please give me free quality professional advice so I can be a baby about it and insult you
Hello,

Getting Webpack, Typescript, and Jest to align with AWS Lambdas' bundle expectations is very annoying. Especially when you want to make sure external libraries are included in the bundle.

That's my post, thanks for reading.

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

Love Stole the Day posted:

Hello,

Getting Webpack, Typescript, and Jest to align with AWS Lambdas' bundle expectations is very annoying. Especially when you want to make sure external libraries are included in the bundle.

That's my post, thanks for reading.

I've never used it but if I were going to start doing Lambdas I would look at creating an image so you can validate that the image you're producing actually has everything in the right place before you deploy it.

madmatt112
Jul 11, 2016

Is that a cat in your pants, or are you just a lonely excuse for an adult?

minato posted:

I'm way out of date, but no they do not. Had a basic coding exercise (read data from 2 files and conjoin them) as the screen, then on the day it was this (literally copied from what the recruiter sent me):

:words:


If I was asked “what is your favorite protocol and why?" in an interview I honestly don’t know what if say. What kind of question is that?
The answer should be “the one that provides the service that’s required” or “the only one of its class”.

I dunno, that question implies some really problematic assumptions about how an engineer should be so married to their work as an identity that they develop something like a “favourite protocol”. Bitch I don’t need to have a favourite protocol to do my job, do it well, and be an excellent professional.

spiritual bypass
Feb 19, 2008

Grimey Drawer
The correct answer is "shoehorn everything into http"

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender

madmatt112 posted:

If I was asked “what is your favorite protocol and why?" in an interview I honestly don’t know what if say. What kind of question is that?
Having an opinion is a signal that you actually think about this stuff. They're not necessarily judging your opinion. I've interviewed countless engineers who claim expertise in $thing, but then when asked what they like / don't like about it, what they'd change, what it's good for and not good for, they just shrug.

Hadlock
Nov 9, 2004

"Name three HTTP codes" or "how does the internet work?" are similar open-ended questions

I just want to know you didn't cram medium.com articles about kubernetes for seven days before your interview, before we go deep on whatever medium.com articles you did cram before the interview.

"how does the internet work" can be really high level, or the guy can talk about the refresh rate of the drivers in the mouse clicking the link, to the v8 javascript engine rendering the link, to tcp/ip, http/s, dns, caching systems, WAF, front end servers, back end servers, then dig into the OS/kernel/multitasking etc

Yes you can robotically recite how to create a hello world deployment on my cluster, what else do you know

Adbot
ADBOT LOVES YOU

LochNessMonster
Feb 3, 2005

I need about three fitty


In this case there is only 1 correct anwser, “one of the earlier wireless protocols”: RFC 1149 - IPoAC

LochNessMonster fucked around with this message at 06:40 on May 1, 2022

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply