|
I used to be VERY into tech and my career advancement and probably would have been a deeply lovely manager of someone who had different priorities. Then I had kids and my priorities completely flipped. I still find the field very interesting and am grateful I get to do this for a living. But I'm not hacking on side projects or reading hours a day of tech blogs outside of work for fun anymore. I'm lucky if I have the energy to play a video game for an hour after the kids are finally in bed. So far I've been fortunate to have managers who are fine with that level of investment. I'm sure it also helped a lot that I had that workaholic drive early on to get me over the Senior+ hump. Maybe once they're older I'll have the time and drive to take on leadership roles again. But for now I am totally cool hanging out as an IC, giving my ~40 hours, and leaving work behind at the end of the day.
|
#
?
Feb 28, 2024 21:00
|
|
|
|
| # ? Jun 4, 2024 23:17 |
|
|
Extremely Penetrated posted:I have 3 copies of externaldns running, one each for Cloudflare, public route53, and private route53. They watch for their own custom annotations on both ingresses & services. I can get a static CNAME in Cloudflare just by making a service of type ExternalName and giving it the same annotation I configured my externaldns-cloudflare provider with. Ah, neat. This isn't well documented, thanks! Kind of wish this kind of "manual override" was supported in annotations, vs having to deploy a weird halfway vegestal service just hanging out there Docjowles posted:I would also say that yeah, if you have never worked with someone like this, you've been lucky or sheltered. Not everyone finds tech fascinating and wants to deeply understand it, or cares about climbing the career ladder. For a lot of people it's just a job like any other and they just want to get their pay check and log off to do whatever is more important to them in life. Agree Two additional thoughts 1. You hired the wrong person, or very likely: 2. They don't want to appear incompetent, and show that as the new guy, they can do basic tasks quickly 2b. You're giving him boring newbie work and he's putting in the appropriate amount of effort
|
|
#
?
Feb 28, 2024 21:01
|
|
|
I'm working on giving my team members better goals in our performance management system this year, and one of the themes I hit on is that I want to recognize the people who are out there learning new stuff, documenting it, and helping us improve. If you just want to show up and do what you're told, that's not awful, you aren't going to be on a PIP or some dumb thing like that, but you should expect your year-end review to reflect that other coworkers are doing more and/or better than you are.
|
|
#
?
Feb 28, 2024 21:49
|
|
|
Hadlock posted:Two additional thoughts I'm probably thinking about this a lot more heavily than I need to, just because of my history, and my desire to do a mentorship. So I'm thinking a lot about how I can mold him. I didn't have any say in the hiring, I think he was interviewed mostly between the time that I did my interviewing and when I actually started, and so he ended up starting only 2 weeks after me. I've got a lot of faith in the hiring process based on what I saw from it, so I don't think it's 1, but I think 2 is very likely. 2b isn't very likely here, because we're having him do terraform from scratch for running our service in ECS (currently running on EC2 instances). Which is, I think, pretty interesting, because he's having to figure out a lot of stuff that's going to be consequential for how we run our product in production eventually. Also it's kind of interesting, the company is hybrid office/WFH, but new hires (like the two of us) are in the office full time for a while, and also our team lead has his plate pretty full, so I'm really trying to deflect all the questions he's going at the team lead with onto me.
|
|
#
?
Feb 28, 2024 21:49
|
|
|
Did you talk to him about (the way he gets to) his solutions? As long as someone’s coachable I really don’t mind people having a way of getting things to work that appears less efficient (or even completely alien) to me. I realize everyone does things their own way and learn things in a way that works for them. I can provide insights in my process and help them understand why I do things. But if they don’t want to or care to hear it, I’m not going to put effort into them either.
|
|
#
?
Feb 28, 2024 22:14
|
|
|
FISHMANPET posted:we're having him do terraform from scratch for running our service in ECS (currently running on EC2 instances). Which is, I think, pretty interesting, because I find writing terraform more dull than watching paint dry; if he's using copilot then maybe he has similar opinions about the drugery he's been assigned. The only reason I'm doing it now is I get to do all the peripheral stuff and designing the larger patterns too that's vastly more interesting
|
|
#
?
Feb 28, 2024 22:56
|
|
|
Hadlock posted:I find writing terraform more dull than watching paint dry; if he's using copilot then maybe he has similar opinions about the drugery he's been assigned. The only reason I'm doing it now is I get to do all the peripheral stuff and designing the larger patterns too that's vastly more interesting Yeah. The act of writing terraform sucks, mostly because HCL is like trying to create precision drawings with a 6 inch diameter novelty crayon. Designing the architecture that you will ultimately express and build with Terraform is fun though. If he's only getting to do the "turn this diagram into code" bit I could see that wearing thin pretty quick. But then again he's a junior potentially doing this all for the first time so you gotta start somewhere.
|
|
#
?
Feb 28, 2024 23:22
|
|
|
Docjowles posted:The act of writing terraform sucks, mostly because HCL is like trying to create precision drawings with a 6 inch diameter novelty crayon. a depressingly large amount of my job is making sure the terraform provider actually does what it is supposed to do and this analogy hits hard
|
|
#
?
Feb 29, 2024 00:21
|
|
|
I'm of the opinion that the problem with terraform isn't hcl itself but literally everything else that happens afterwards
|
|
#
?
Feb 29, 2024 00:22
|
|
|
FISHMANPET posted:At my last place we hired someone who came from an MSP, and he was great, because his whole career had been "no documented process, figure it out" There's a third kind of engineer, "I can't figure out your system/it's so opaque I don't understand it, so I'm going to rewrite this system instead" which arguably might be the worst
|
|
#
?
Feb 29, 2024 02:31
|
|
|
Hadlock posted:There's a third kind of engineer, "I can't figure out your system/it's so opaque I don't understand it, so I'm going to rewrite this system instead" which arguably might be the worst Close. Having worked with “this system does not conform exactly to my prior experience so it sucks and I am going to rewrite it on my preferred tech stack even if doing so makes no sense and adds no value” I would argue that is actually the worst.
|
|
#
?
Feb 29, 2024 02:38
|
|
|
Docjowles posted:Close. Having worked with “this system does not conform exactly to my prior experience so it sucks and I am going to rewrite it on my preferred tech stack even if doing so makes no sense and adds no value” I would argue that is actually the worst. The bestest yet is when you get a person that has this mentality while being considered the tech anchor on the team and they just go off and cowboy up on their own and demand that the poo poo they did somehow gets realized into actual work streams.
|
|
#
?
Feb 29, 2024 02:48
|
|
|
Extremely Penetrated posted:I think your problem is the need to look up that S3 bucket URI and patch it to the service. Do you really need to create/destroy S3 buckets on the fly? If you can have static bucket URIs for your CNAMEs then your problem goes away. Otherwise, this is where ACK falls on its face and you have to use a Crossplane XRD (which supports arbitrary patching of attributes from one resource to another). This is something that's dead simple in terraform/cloudformation but is pretty poo poo in kubernetes. Like I wanted to create EC2 instances and then use their new private IPs for target group registrations as well as private route53 records, and a Crossplane XRD was the only decent option I found. You're the only person to reply to my cloudfront thing so just dragging this out as long as possible So I have this complex* thing that spits out the updated gitsha in yaml of every branch on container build to a git repo that gets consumed to feed the Argo monster for helm values Kind of thinking that on initial container build it also spits $branchname into a second line in the yaml $ifnotexists and then point the cloudfront distribution ACK at that bogus value. It'll fail out on the k8s side, but won't cause the helm deploy to fail since a value exists where it needs one Then I have an init container that polls $s3bucketACK_ARN where ARN != $branchname, then patches (via I just invented flux for S3 CDN The Aristocrats!  *Actually it's like 3 lines of bash Hadlock fucked around with this message at 04:21 on Feb 29, 2024 |
|
#
?
Feb 29, 2024 04:12
|
|
|
Hadlock posted:I just invented flux for S3 CDN The Aristocrats! The thread title is already impossibly long but I love the idea of it ending in an Aristocrats joke
|
|
#
?
Feb 29, 2024 05:56
|
|
|
The more I think about it, it makes sense The DevOps Thread: There, I fixed it, The Aristocrats!
|
|
#
?
Feb 29, 2024 17:03
|
|
|
Question for the room: I distinctly remember “never ever ever run a database in a container” being common wisdom back in the mid 20teens. That’s clearly no longer a problem and I find myself wondering if that was ever an actual issue and things have changed, or it was a misunderstanding of the tech/an old engineer’s tale. What’s the word there?
|
|
#
?
Mar 1, 2024 16:24
|
|
|
Warbird posted:Question for the room: I distinctly remember “never ever ever run a database in a container” being common wisdom back in the mid 20teens. That’s clearly no longer a problem and I find myself wondering if that was ever an actual issue and things have changed, or it was a misunderstanding of the tech/an old engineer’s tale. What’s the word there? I instinctively doubt all "never ever evers" because there are niche use cases for all sorts of stuff. I know I was pushing hard against DBs in containers when the DBAs I support first asked about it because I correctly doubted that they understood the container stack. They just thought containers were a new type of VM. I'd ask what persistent storage model they had in mind and how it would interact with our infrastructure, and they'd be surprised they were even expected to know the answer to that question, because they didn't manage storage. Even back then, though, there were always people reporting success in specific use cases.
|
|
#
?
Mar 1, 2024 16:49
|
|
|
This was at the start of my career at the time and the person telling us was the resident “reads white papers for fun” graybeard rear end in a top hat so we were expected to not question. It wouldn’t surprise me if he was always wrong or was giving us incomplete/bad info as he tended to do.
|
|
#
?
Mar 1, 2024 16:52
|
|
|
Extremely Penetrated posted:I think ACK support from AWS got yanked hard. Once my TAM got wind that I was even considering using EKS they started having Come to Jesus interventions with me, and even put together a call with 3 other AWS specialist support engineers to try to dissuade me. (They didn't offer a better solution, just a nebulous "automate Cloudformation!") It's cynical of me but I suspect they have orders from on high to discourage kubernetes use however they can, to make sure you're locked in to the AWS-specific services. Maybe, but as a former aws engineer (on the build services side, not the handhold customers side), kubernetes' popularity was just baffling. It really does/did look to us like a way of spending more money (sometimes a LOT more money, so many people ignored cross-AZ bandwidth costs) on aws, in order to get a worse result. I would legitimately believe they've transitioned from "lol our customers are demanding to spend more money haha give them what they want" to "oh god, the kubernetes-shaped money firehose they built is getting so expensive they're starting have a bad experience with 'the cloud' in general, we don't want them to end up motivated to gtfo, let's help them get their poo poo together even if it severely reduces their spend short term"
|
|
#
?
Mar 1, 2024 16:53
|
|
|
databases aren't stateless machines that you can turn off at will, or scale up to instantly get more capacity, so if you're using containers for them you should ask what you're actually getting out of the container stack. they also tend to be critical pieces of infrastructure when they exist, so you should do your normal due diligence of cost and risk modeling. if you do this, most of the time you'll find that running database workloads on top of a container platform is extra risk and extra cost for 0 reward. crazypenguin posted:It really does/did look to us like a way of spending more money (sometimes a LOT more money, so many people ignored cross-AZ bandwidth costs) on aws, in order to get a worse result.
|
|
#
?
Mar 1, 2024 17:06
|
|
|
Warbird posted:Question for the room: I distinctly remember “never ever ever run a database in a container” being common wisdom back in the mid 20teens. That’s clearly no longer a problem and I find myself wondering if that was ever an actual issue and things have changed, or it was a misunderstanding of the tech/an old engineer’s tale. What’s the word there? Some databases were built with containers in mind, but they are all relatively new so I don’t trust them yet for any serious workloads. Every other database has support bolted on well after the fact. We have/had efforts here to run databases on K8s and the first time an underlying host gets rotated out or patched the DBA or owner immediately comes and complains and requests an instance that never gets retired, or 2 weeks heads up on a maintenance and we just tell them too bad. The big problem is that DBAs and to a lesser extent feature teams don’t understand the implications and often don’t like to read documentation that goes against their beliefs. This means they get surprised in a bad way and then you have to have 2 months of meetings with PMs to discuss how to accommodate or migrate away.
|
|
#
?
Mar 1, 2024 20:01
|
|
|
K8s does have support for stateful sets even though it's a platform designed for stateless services. You could in theory run Postgres container on it's own node/node group and you're most of the way to building home-spun RDS on k8s. If your app needs the DB to store and update smaller amounts of data, like sub 4gb I think I'd be ok running it on a container? Especially in staging or ephemeral environments. The biggest Postgres instance I've regularly used on a container was 10gb Putting 100gb+ production database that the entire company runs on and is dependent for revenue generation? Use the right tool for the job. It probably takes less time and effort to spin up and administrate an RDS server than building the rube goldberg machine that is stateful Postgres on k8s. George Wright posted:and often don’t like to read documentation that goes against their beliefs. Just wanted to highlight this beautiful part of your post 
|
|
#
?
Mar 1, 2024 20:45
|
|
|
I haven’t had terrible experiences running vendor dependent databases on k8s Postgres, but I haven’t had great experiences with it either! You almost certainly want to use one of the Postgres operators like https://cloudnative-pg.io to do so, if only to make updates less agonizing. but it’s also really not a well suited tool for the job, and I wouldn’t recommend it in the slightest for anything real. at the end of the day you only have one primary pod which is going to go down every time you update k8s versions. It’s a far cry from the experience you get using a managed DB service or, honestly, running it on VMs directly.
|
|
#
?
Mar 2, 2024 00:32
|
|
Cat Army
|
My static environments use RDS, but my ephemeral environments each have a mysql container backed by EFS. Microservices run their sql scripts as init containers, which adds a lot of startup time but guarantees their schemas are correct. It was relatively painless to set up and hasn't given us much trouble. Cleaning up expired environments from EFS was a few more lines in the Purge cronjob.crazypenguin posted:Maybe, but as a former aws engineer (on the build services side, not the handhold customers side), kubernetes' popularity was just baffling. That's probably a more realistic view than mine, I'll try to give them the benefit of the doubt. Honestly the biggest factor for using EKS vs ECS for these ephemeral environments was the available tooling / user experience. My devs have had access to the ECS UI for years and still get lost in there. They are better with Docker Desktop, so I wanted to present them with something like that (or Podman). ArgoCD's UI isn't about to win any awards but it does a much better job of showing you your containers, their health & logs, and letting you change crap like environment variables on the fly.
|
|
#
?
Mar 2, 2024 08:07
|
|
|
the best cloud bullshit YouTuber out with yet another banger https://www.youtube.com/watch?v=ia8Q51ouA_s
|
|
#
?
Mar 2, 2024 19:16
|
|
|
I still want to know when Omegastar will support ISO dates (there was a video from November 2021 where it was delayed again).
|
|
#
?
Mar 2, 2024 19:54
|
|
|
FISHMANPET posted:I still want to know when Omegastar will support ISO dates (there was a video from November 2021 where it was delayed again). I was working with a junior on my team and told him "Anytime someone tells you in software development that there is only ever one right answer to a problem, they're absolutely full of poo poo, with one exception. ISO-8601 is the only acceptable datetime string format, and anyone who says otherwise is a terrible person with brain worms. But yeah everything else has multiple answers." I'm right.  The Iron Rose posted:the best cloud bullshit YouTuber out with yet another banger Dunno if this is common knowledge, but he works at one of the big FAANG companies, which explains an awful lot. I know a few people who know him IRL. This latest video felt like he was spying on me.
|
|
#
?
Mar 2, 2024 20:15
|
|
|
The Iron Rose posted:the best cloud bullshit YouTuber out with yet another banger KRAZAM is great. "Delivering this feature goes against everything I know to be right and true, and I will sooner lay you into this barren earth than entertain your folly for a moment longer!"
|
|
#
?
Mar 2, 2024 21:04
|
|
|
I talked to Kelsey Hightower at DevOps Days back around 2016 and he was talking about why particularly RDBMSes are a bad fit for most containerization use cases even with the existence of StatefulSets. This was in the context of people asking whether they should suggest MS SQL and Oracle for containerization efforts in their company. Some people were getting a bit anxious and were pushing back against him when it was clear to me anyway that Kelsey had actually tried it before with examples of failure modes encountered and these folks hadn’t. Now it’s 2024 and there’s K8S operators and more stable K8s implementations of everything under the sun now, so it may be better, but in practice what I think is the real reason to avoid containerizing them isn’t because of necessarily technical reasons in most companies but bureaucratic / political ones. Companies still have separate database architects and such, and IME they tend to really, really dislike adding any more layers of complication to their stacks.
|
|
#
?
Mar 4, 2024 00:38
|
|
|
There's basically no benefit. Containers make deployment faster, which you don't often want to do with a database until you reach a certain level of sharding. Deploying quickly, even to track patch releases, comes with its own drawbacks: restarting a database flushes the in-memory cache, and your performance probably suffers from cold starts. Container tech has gotten a lot better in the last decade, but there's still overhead. The one place in your stack you might really need to max your vertical scale is probably the worst spot for it, especially if you're licensing your database tech by the CPU. It's not like this type of isolation confers many advantages on a system that isn't running multiple workloads in the first place.
|
|
#
?
Mar 4, 2024 13:49
|
|
|
Sorry this is kinda offtopic but I don't follow the generic IT bitching threads. Are the rest of you blessed with security teams that don't make you want to start drinking at 10am? Had the following interaction today with someone whose literal job is cloud security "Hey can you help me figure out why my AWS load balancer won't work? I am trying to hit https://stuff.whatever:1234 probably the firewall is blocking it???" "Uh well first of all there is no firewall in this path. Second, your LB only listens on port 443, which is probably good! Your target group sends it to the backing instances on port 1234 which is also good. You shouldn't include that in the URL though." "Cool thanks I changed that it still doesn't work though" "Well the target group is failing its health check. I see that the security group on your instances doesn't allow the load balancer to hit 1234 so that's your problem" "Cool thanks I will just add a rule to allow 0.0.0.0/0, cya" Motherfucker your own team has a tool that scans our AWS accounts and spams nasty Jira tickets if it detects a 0.0.0.0/0 rule. I assume they've somehow exempted themselves from it, would be extremely on-brand. These kinds of questions that show a fundamental lack of understanding of how things like load balancers and security groups and DNS work, from the cloud security team, are disappointingly common. I hate to crap on anyone just trying to do their job since I certainly ask and do dumb stuff all the time. But "I would simply change the security group to allow all" is the kind of thing that should fail an initial phone screen 
|
|
#
?
Mar 4, 2024 18:53
|
|
|
I don't have any real conflicts with security. My biggest pain comes from app devs who are just the laziest motherfuckers in the world and will do the dumbest possible poo poo in order to avoid anything that looks like a hurdle.
|
|
#
?
Mar 4, 2024 19:03
|
|
|
infosec/secops/etc has been a fake discipline for like a decade now, at this point. there's 1 useful type of person you can hire in this dimension and its "compliance architect". it's the person you go to when you have a question about what controls need to be met by <thing>, how the controls are usually satisfied, what the audit chain is, what organization certifies us, what the business need for the certification is, what happens if we don't do something we should, and how we prove that we actually do the things we need to. they go to meetings that you aren't involved in to argue with lawyers and c-levels about what the definition of words like "access" is. their job is to get the company certified for things, which increases the company's customer base and ideally also its revenue, without being an overall ROI detriment to the engineering organization. every other job role in this field is make-work.
|
|
#
?
Mar 4, 2024 19:05
|
|
|
Our security managers have been decent, and are happy to help me gang-tackle developers and application engineers who ignore best (or even third -best) security practice. Some of our security engineers are just wretched, though. One keeps sending out vulnerability reports with boilerplate text saying "please find vulnerability details and affected servers attached." He never attaches the list. I have to ask every time. Sigh.
|
|
#
?
Mar 4, 2024 19:26
|
|
|
Docjowles posted:Sorry this is kinda offtopic but I don't follow the generic IT bitching threads. Are the rest of you blessed with security teams that don't make you want to start drinking at 10am? Current cloud sec team is pretty good but extremely understaffed. They put me in contact with the team that manages/builds guardrails and checks so I can work with them to improve them. They just ask me to keep them in the loop so they have an overview on what gets done so they can use it as a reference architecture / reusable pattern. Last gig had a security architect who literally only attended meetings and told other people they were doing it wrong. He never did or delivered anything. There was no operational security team. Gig before that had a cloud security team with 1 person embedded per business unit. Our guy was a gatekeeper and never shared anything. Refused to automate or document things so he was the only one who could check and change things. The result was a horrible mess and he kept screaming he was to busy putting out all the fires (which he was creating himself). He either had dirt on the CEO or they were afraid he wouldn’t turn over the AWS account credentials if he got fired.
|
|
#
?
Mar 4, 2024 19:29
|
|
|
12 rats tied together posted:every other job role in this field is make-work. This hits hard. I swear their full time function is doing POC's of new tools we don't end up using but still require work from other teams to properly integrate and evaluate. Or in the rare case they do adopt one, switching to a new one 6 months later that does the same basic thing but needs people to change a bunch of stuff in order to conform to it. Doing this busywork for them is mandatory, of course, because security.
|
|
#
?
Mar 4, 2024 19:57
|
|
|
Not to pick more fights but speaking of title inflation/invention I noticed we have a "data engineering" thread I guess I've just never dealt with a company that generates more than 100gb of data per week, but at what point does data/analytics engineering become something fully independent of devops. This task always seemed very much "operations" side of "DevOps" to me please educate me
|
|
#
?
Mar 4, 2024 20:15
|
|
|
Docjowles posted:This hits hard. I swear their full time function is doing POC's of new tools we don't end up using but still require work from other teams to properly integrate and evaluate. ... Doing this busywork for them is mandatory, of course, because security. I've so far managed to nope out of this work. This largely seems to be other people's attempt to claw their way into management as POCs are the only way to get on director level and radar typically Goddamn I need another cup of coffee my cynical level is off the chart this morning
|
|
#
?
Mar 4, 2024 20:18
|
|
|
Hadlock posted:Not to pick more fights but speaking of title inflation/invention I noticed we have a "data engineering" thread Data engineering usually means building/maintaining a data What makes it different is loading data into it from dozens to thousands of sources (ranging from oracle to db2 and csv’s to SaaS API’s. You’ll be doing ETL on the data and keeping it up to date. Once it’s in there’s usually hordes of data scientists wrangling information out of it with mostly bad python “models” and worse sql queries. Someone needs to make sure all of that is working properly and data is accessed only by people who are supposed to. Often you’d also like to manage things like meta data, data lineage, etc.
|
|
#
?
Mar 4, 2024 23:19
|
|
|
|
| # ? Jun 4, 2024 23:17 |
|
|
So data security isn't a devops problem any more? I just throw it over the wall to the
|
|
#
?
Mar 5, 2024 00:22
|
|