|
 Yesterday my coworker proposed moving one of the major dev environments databases from an old postgres handbuilt thing to Amazon RDS. We're on PG 9.5 in prod. He suggested just upgrading to RDS pg 11 straight away. We spent the next 20 minutes in a meeting with 15 engineers + release manager telling him no, even though pg 11 is technically compatible, it's not a good idea to jump straight to 11 immediately. He finally relented and we all agreed RDS was fine, but only use pg 9.6, as it's a safe known version for our monolith. Today in another meeting with 30+ engineers in it, he announces that he converted the major dev environment to RDS pg 11 anyways. So we again, ask him what version production is on (9.5), how can we be certain what we're testing on pg 11 is going to catch problems with our ORM between pg 9.5 and 11, exact same conversation as not even 18 hours before. Finally the software architect gets involved in the conversation tells him no and he agrees again not to convert any more major environments to pg 11 and stick to pg 9.6. Hadlock fucked around with this message at 04:55 on Apr 16, 2020 |
#
?
Apr 16, 2020 04:53
|
|
|
|
| # ? May 15, 2024 23:58 |
|
|
What a clueless guy. Amazon RDS has been offering PG 12 for two weeks already 
|
|
#
?
Apr 16, 2020 19:15
|
|
|
Hadlock posted:
So by now all environments are on pg 11?
|
|
#
?
Apr 16, 2020 19:35
|
|
|
Hughlander posted:So by now all environments are on pg 11? Either that or everything is now running on Aurora. 
|
|
#
?
Apr 16, 2020 19:40
|
|
|
Is there a valid, non-lovely reason, why bitnami now owns most of the critical helm charts The optics of a for profit company owning a bunch of critical helm charts, which directly competes with their own product, does not seem great
|
|
#
?
Apr 16, 2020 22:46
|
|
|
The troll answer is to make you nervous so you vendor them yourself. Real answer is probably no good reason other than some internal CNCF agreement
|
|
#
?
Apr 16, 2020 22:55
|
|
|
Hadlock posted:Is there a valid, non-lovely reason, why bitnami now owns most of the critical helm charts I'm seriously FURIOUS at them for unilaterally coming in, forcibly taking over the Postgres chart, pissing off a few long time contribs in the process and breaking it. Oh yeah then once they started doing things their way they deprecated helm/postgres and won't approve any PRs into it anymore, and now want you to use their bitnami/postgres chart. gently caress Bitnami that poo poo is community hostile. tl;dr they took over a chart and told everyone using it to go gently caress themselves.
|
|
#
?
Apr 17, 2020 02:35
|
|
|
SeaborneClink posted:I'm seriously FURIOUS at them for unilaterally coming in, forcibly taking over the Postgres chart, pissing off a few long time contribs in the process and breaking it. Not knowing much about helm is it something to just fork and direct everyone to the version maintained?
|
|
#
?
Apr 17, 2020 02:55
|
|
|
Kind of, but that starts to defeat the purpose of helm. We've been mirroring the official repo for the past year or so, but we vendor most of our charts like the other goon mentioned. So less of an impact there. Agreed that it sucks for all involved.
|
|
#
?
Apr 17, 2020 03:05
|
|
|
helm is fine to fork since if you're using it for anything more complicated than the 'yum install postgres' of kubernetes you are going to be writing your own charts anyway
|
|
#
?
Apr 17, 2020 05:01
|
|
|
12 rats tied together posted:helm is fine to fork since if you're using it for anything more complicated than the 'yum install postgres' of kubernetes you are going to be writing your own charts anyway
|
|
#
?
Apr 17, 2020 20:39
|
|
|
The Fool posted:Also known as “the real world” for a ton of enterprises Vulture Culture posted:Yeah, I've never been able to run a Helm chart in production without modifying something about it Flux seems to work better, at least for our needs. Bhodi fucked around with this message at 22:10 on Apr 17, 2020 |
|
#
?
Apr 17, 2020 22:06
|
|
|
Bhodi posted:I've been screaming for 20 years and haven't stopped screaming yet I hate helm, I just joined a new place that’s all in on it and the amount of bullshit they do to use helm is terrifying. The entire concept of a release failing on k8s and leaving you in a broken state is totally the opposite of a convergent api k8s is. It theoretically makes cleanup easier but that’s not worth it, not when it will just bomb out at the slightest touch.
|
|
#
?
Apr 17, 2020 22:10
|
|
|
Bhodi posted:Our k8s lead absolutely hates helm ...And then there's tiller. That drat thing just will not cooperate long-term, it's almost magical how it ends up able to stop itself in various ways. Helm 3 drops the need for tiller. At that point it really is just a templating engine, along with plugin support for sops (helm secrets).
|
|
#
?
Apr 17, 2020 22:18
|
|
|
helm is alright if you ignore third party charts and use it to clone stamp YAMLs to multiple clusters, or in slightly different ways in the same cluster. like cluster autoscaler needs to be told about different ASGs for the infra, staging, and prod clusters. that works fine. what is terrible are the everything-you-need charts full of blocks like this:code:
|
|
#
?
Apr 17, 2020 23:03
|
|
|
My favorite thing is when the chart is 40% " thing | .toYaml". I would rather just write the yaml.
|
|
#
?
Apr 17, 2020 23:12
|
|
|
I like code reuse and I cannot lie You other ops guys can't deny That when a junior with a little K8s And a hard-on for Helm gets hired You give notice, want to stab your face 'Cause you don't want to maintain that
|
|
#
?
Apr 17, 2020 23:49
|
|
|
necrobobsledder posted:I like code reuse and I cannot lie Did you recently transition a bunch of postgres db to RDS by any chance
|
|
#
?
Apr 18, 2020 00:29
|
|
|
freeasinbeer posted:I hate helm, I just joined a new place that’s all in on it and the amount of bullshit they do to use helm is terrifying. The entire concept of a release failing on k8s and leaving you in a broken state is totally the opposite of a convergent api k8s is. Nomnom Cookie posted:helm is alright if you ignore third party charts and use it to clone stamp YAMLs to multiple clusters, or in slightly different ways in the same cluster. Nomnom Cookie posted:the only way I've found to consume a third party chart and stay sane is to copy it locally and rip out all the stuff that doesn't make sense for your environment and makes the chart illegible Helm is loving trash The only thing worse than helm is helmfiles. Absolutely never again. Methanar fucked around with this message at 00:49 on Apr 18, 2020 |
|
#
?
Apr 18, 2020 00:46
|
|
|
Hadlock posted:Helm 3 drops the need for tiller. At that point it really is just a templating engine, along with plugin support for sops (helm secrets). it also enables rollouts on configmap changes! (kustomize also has that, do not use helm) Nomnom Cookie posted:helm is alright if you ignore third party charts and use it to clone stamp YAMLs to multiple clusters, or in slightly different ways in the same cluster. 12 rats tied together posted:My favorite thing is when the chart is 40% " thing | .toYaml". I would rather just write the yaml. kustomize, kustomize, kustomize
|
|
#
?
Apr 18, 2020 00:47
|
|
|
it's literally me, the "k8s lead" who... alright, i won't say i hate helm, but my experiences with it made me instantly dubious and rapidly alarmed.
|
|
#
?
Apr 18, 2020 00:52
|
|
|
Still makes me puke a little in my mouth to keep churning out more and more YAML for... everything. Then I look at the 3k+ LOC of Jenkins I have to maintain and remember things could be worse Hadlock posted:Did you recently transition a bunch of postgres db to RDS by any chance
|
|
#
?
Apr 18, 2020 01:50
|
|
|
We had a massive blowup between leads on two different teams so somehow all our kube clusters wound up with weave works and helm (controlled by Terraform) managing different things. Our usecase for kube is so simple that we don't even really need anything more than flux, so now that both those leads are gone we're going through and consolidating everything we possibly can.
|
|
#
?
Apr 18, 2020 02:45
|
|
|
Doc Hawkins posted:kustomize, kustomize, kustomize Kustomize is bad too but at least it is built into kubectl. It's probably the least offensive of the toolchains people actually use, but it's still just a worse ansible. A coworker wrote our ansible kustomize plugin and its fine for what it does but roles that use normal ansible are just normal looking ansible, and roles that use the kustomize plugin have an extra 28 subfolders and kustomization.yamls in them for 0 extra features than what normal ansible does by default.
|
|
#
?
Apr 18, 2020 06:40
|
|
|
Methanar posted:Helm is loving trash You sound like someone who tried to use a third party chart
|
|
#
?
Apr 18, 2020 07:02
|
|
|
12 rats tied together posted:Kustomize is bad too but at least it is built into kubectl. 28 is a lot. a typical service for us will have one per environment, plus one shared base. i'd consider ansible for its ubiquity, but it does ten thousand things we wouldn't use.
|
|
#
?
Apr 18, 2020 08:24
|
|
|
Doc Hawkins posted:i'd consider ansible for its ubiquity, but it does ten thousand things we wouldn't use. This is a totally fair decision to make, IMO. We use it for everything from creating and managing our ... 9? kubernetes clusters down to managing WAN interconnects between various permutations of public cloud provider, account, and region with the physical DCs that we manage, so it is the perfect choice for us. The subfolders mostly come from namespaces, which we have a lot of, and which have a many-to-many relationship between clusters. We have a base set of namespaces that must exist on every cluster, and then we have per-cluster addon namespaces, and then we're managing least-privilege RBAC for each of these namespaces-in-clusters, alongside your usual assortment of cluster addons: kube2iam, external dns, etc. The mapping between our cluster dns config and the actual dns recursors that respond to DNS requests from each cluster is also maintained in this codebase, since we have a number of applications running that would absolutely destroy an AWS VPC DNS server. This ended up being really handy when someone deployed a new app-specific cluster that inherited the default dns config and then started making hundreds of thousands of DNS queries per second. Being able to recognize this problem and then make the entire set of changes required to address it (new dns recursor pool, the new pool has distinct configuration optimized for the application behavior, the new pool's routing to/from the cluster nodes, the cluster's configuration that causes it to target the new pool) in a single workflow, with a single reviewer, and a single ansible-playbook is a good example of what ansible-for-k8s can do you for that isn't just "make it so I don't have to write so much yaml". Ideally, though, you don't have to do any of that poo poo and your employer is still making money and can keep paying you. We still have some folks internally who don't want to be involved with the entire infra ecosystem and just want a little fiefdom where they can type kubectl -k or helm upgrade instead of ansible-playbook. It's annoying to deal with, especially when they eventually run into issues that would be solved automatically for them if they were using the shared codebase, but ultimately I'm just an IC and I can't tell them how to do their jobs.
|
|
#
?
Apr 18, 2020 19:46
|
|
|
Nomnom Cookie posted:helm is alright if you ignore third party charts and use it to clone stamp YAMLs to multiple clusters, or in slightly different ways in the same cluster. like cluster autoscaler needs to be told about different ASGs for the infra, staging, and prod clusters. that works fine. what is terrible are the everything-you-need charts full of blocks like this: This always infuriated me about working with Chef. The Chef company and community REALLY want you to use the community cookbooks whenever possible. But they inevitably end up growing to support every possible feature and use case of the thing they manage no matter how obscure or insane. Plus you have to get all the dependent cookbooks whether you use their features in your environment or not. Have no windows machines? Well too bad this software might possibly run on windows so now you have to drag in the windows cookbook. Etc. They also love to completely rewrite them in totally backward incompatible ways every six months. We nearly always end up forking them if not outright writing our own that does the three things we needed in 10,000 fewer lines of code. So it’s pretty hilarious that all this bullshit has been dragged forward into the world of containers.
|
|
#
?
Apr 19, 2020 13:53
|
|
|
Docjowles posted:This always infuriated me about working with Chef. The Chef company and community REALLY want you to use the community cookbooks whenever possible. But they inevitably end up growing to support every possible feature and use case of the thing they manage no matter how obscure or insane. Plus you have to get all the dependent cookbooks whether you use their features in your environment or not. Have no windows machines? Well too bad this software might possibly run on windows so now you have to drag in the windows cookbook. Etc. They also love to completely rewrite them in totally backward incompatible ways every six months. We nearly always end up forking them if not outright writing our own that does the three things we needed in 10,000 fewer lines of code. A year and a half ago I had some non-sense bug that broke production in a nasty way that I can't remember the details of at all but was caused by a windows cookbook thing not working properly in GCE the way it did without a cloud environment. That's my entire experience of working with windows for the last 4 years.
|
|
#
?
Apr 19, 2020 18:22
|
|
|
Docjowles posted:This always infuriated me about working with Chef. The Chef company and community REALLY want you to use the community cookbooks whenever possible. I'm always quick to point out something that ansible does better than chef or whatever other product, but in this case ansible galaxy is as exactly as dogshit as the chef marketplace. Sourcing your automation code from a community location is just a bad pattern in general unless you either have the absolute simplest requirements possible in your provider or unless you have time to comprehend and support 10x as many lines of code as are required to solve your problem. Whatever the terraform one is called sucks rear end too. My experience working with windows in the past 4 years is that there is still apparently a gestalt opinion that powershell can solve every windows problem so we have a 6 digit LoC powershell commitment that doesn't have things like "a module" or "functions with return values". I don't think anybody is actually doing windows infrastructure well at the 5000+ node range except maybe Microsoft? It's also really really hard for us to find windows infra engineers that aren't way worse candidates than "generic aws engineer who spent a few hours reading MSDN documentation". I'm sure they exist but they aren't applying here.
|
|
#
?
Apr 19, 2020 18:40
|
|
|
We went down the same road with puppet. gently caress yeah, community modules! Never write code again! Until we realized many modules are more complicated to use than hand writing config files for the software we're using and stop getting development or don't work right or get refactored regularly. So now we've gone back to storing config files in puppet with limited use of templates when we really need per-server customizing.
|
|
#
?
Apr 19, 2020 20:23
|
|
|
12 rats tied together posted:Whatever the terraform one is called sucks rear end too. I hate these things https://github.com/terraform-aws-modules/terraform-aws-security-group
|
|
#
?
Apr 19, 2020 20:39
|
|
|
Keep the types/providers/etc. from the community repositories and discard literally the entire rest. I want libraries that make it easier to implement my own opinions of what good looks like, not harder.
|
|
#
?
Apr 19, 2020 20:51
|
|
|
12 rats tied together posted:Sourcing your automation code from a community location is just a bad pattern in general there it is. helm is probably not especially bad, especially now that it has no server component. it was my first serious encounter with this idea.
|
|
#
?
Apr 19, 2020 22:49
|
|
|
We use a handful of Puppet modules from the community, but we always try to get a feel for who the author is and what level of commitment they have to the project. If it's some rando who just wanted to solve a problem and walk away, we either fork it ourselves, or steer clear. When it's someone who makes a living as a Puppet consultant, or has been making frequent commits for years and seems to be on top of things, then it's good to go.
|
|
#
?
Apr 19, 2020 23:15
|
|
|
I use helm to install the nginx ingress controller and the Prometheus operator on our clusters, but that’s it. I tried to justify using it for our own apps but it just seems like a huge fuckin’ hassle.
|
|
#
?
Apr 20, 2020 03:49
|
|
|
SeaborneClink posted:Oh yeah then once they started doing things their way they deprecated helm/postgres and won't approve any PRs into it anymore, and now want you to use their bitnami/postgres chart. Everyone is doing this because the helm/charts repo is going away after November. Good riddance IMO, that central repo was a pain to deal with for development.
|
|
#
?
Apr 20, 2020 04:07
|
|
|
Methanar posted:I hate these things https://github.com/terraform-aws-modules/terraform-aws-security-group
|
|
#
?
Apr 20, 2020 04:19
|
|
|
Terraform modules that try to account for every possible permutation of the thing they're encapsulating end up being more of a pain in the arse to use and troubleshoot than if you just defined all the resources yourself. I feel like modules make way more sense when you're defining all the infra for one of your applications in it and then reusing it across environments. That VPC module is probably one of the exceptions because there's basically a set-in-stone best practice for almost every part of a VPC you're going to configure.
|
|
#
?
Apr 20, 2020 14:11
|
|
|
|
| # ? May 15, 2024 23:58 |
|
|
Hopefully this is a good place for this and isn't a dumb question. My organization is looking at handing out Azure subscriptions inside our Enterprise Account to smaller sub-units within the parent organization. There's a whole checklist of steps that have been getting done manually, which I'd like to automate. (Things like giving our billing software access, setting up VNETs and routing to the rest of the organization, and so on). Some options: Write some kind of batch script wrapping Azure CLI, and just throw it in Jenkins or something. Advantage: Simple, az CLI is pretty good/reliable. Disadvantage: Seems kinda hacky. Integrate it into our in-house deployment tool/portal using Azure REST APIs. Advantage: Cleaner for end users, easier to delegate its use to existing portal users. Disadvantage: Azure REST APIs are poorly documented and not always full featured. Use Azure Blueprints or ARM templates. Advantage: Native way of doing this? Disadvantage: Vendor-specific way of doing things, and also still have to execute them within the portal I think? Use Terraform. Advantage: Flexible, powerful, clean. Disadvantage: Still gotta figure out where to run it? Jenkins? I'm leaning towards Terraform, my other question was around how Terraform maintains state. Is Terraform a poor fit when this is only going to be used for initial setup, and not on-going maintenance, etc.?
|
|
#
?
Apr 21, 2020 03:32
|
|