|
Blinkz0rz posted:Hot take: deploying kubernetes (properly) and maintaining deployment systems on top of it takes more work (and reaps less rewards) than a mostly working existing system. Of course, we're not deploying kubernetes properly, we're literally throwing together whatever we can get into production as fast as possible because we were given 4 months to go from 0 to fully deployed, with December being one of those months. So it's a toss-up as to how well it actually performs. We already ran into this unresolved issue running on m5s: : https://github.com/kubernetes/kubernetes/issues/49926 and so I look forward to stepping on other land mines.
|
#
?
Dec 19, 2018 01:56
|
|
|
|
| # ? May 15, 2024 04:33 |
|
|
I do admit that applications developed from the get go in containers will likely follow stateless, 12F-ish patterns throughout its lifespan. Stuffing secrets into container layers? Fine, we’ll rip it out later when we go through audits. Writing transaction state to disk? Too bad it’ll get blown away on a new deployment - have fun on call. Want to scale out more? At least you’re less likely to try to write an abortion of 1200+ threadpools bloating out to 12 GB of RAM for no reason instead of launching containers to scale around the entities that are better representative of client demand. It turns out our software launches a new thread pool for every queue that it listens to and over time it’s listening to 1100 queues. Making Bad Things harder to do is always a Good Idea to me. Most software on traditional, state fully managed OSes gives way, way too much freedom to do actively harmful patterns that don’t add business value and that developers usually give no fucks about. For everything else, there’s Operators.
|
|
#
?
Dec 19, 2018 02:02
|
|
|
minato posted:We've used Kubernetes for years and haven't felt the need to automate anything with Ansible (or Chef, Puppet, etc). We use a combination of Jenkins to monitor our gitops repos that contain the kubernetes manifest files, which in turn triggers Helm/Tiller re-deployments. It works very well for 95% of the apps we run. We use AWS RDS for databases and EBS for persistent storage (which Kubernetes supports). I think you'd have to be crazy to seriously endorse doing anything related to container scheduling with Chef or Puppet. For your use case ansible is a drop-in replacement for Jenkins -- whether or not that's good for you pretty much just depends on how much you guys hate jenkins. necrobobsledder posted:I do admit that applications developed from the get go in containers will likely follow stateless, 12F-ish patterns throughout its lifespan. It's not like you can't (and people haven't) been doing all of this _without_ containers for the past 6? 7? years though. It's unfortunate that k8s is what it took to get development shops as a whole to embrace 12 factor instead of, I dunno, 12 factor?
|
|
#
?
Dec 19, 2018 03:39
|
|
|
That's the thing - developers under pressure in a sprint-based org will adopt "worse is better" and to bang out their stories start making decisions to favor laziness now and kicking design cans down the road, so your framework / architecture that outright blocks them from making designs that don't punish them (and the business) immediately will be swept under the rug as technical debt. It's ok to do that if it buys you something pretty nice like for a demo that buys your company some runway, more investors, etc. but after being a long-time bitch for cleaning up the technical debt of application teams of Christmas Past it's clear to me the patterns of laziness occur when developers are given shortcuts regardless of how smart they are. I just can't make up for dozens and dozens of decisions that result in mass outages due to trying to update things faster (per business needs) that are very, very resistant to ever being restarted by design. I am typing this literally during an outage that would have been solved if we had just let services die in a container when threads are exhausted and spun up new ones to take things over. I am writing a script to remove an instance from an ALB, keep it in there for 5 seconds, remove it, and restart the service which takes 2 minutes. So we need a couple hundred or so instances to serve a piddly rate of 300 requests / second. necrobobsledder fucked around with this message at 04:29 on Dec 19, 2018 |
|
#
?
Dec 19, 2018 04:27
|
|
|
12 rats tied together posted:For your use case ansible is a drop-in replacement for Jenkins -- whether or not that's good for you pretty much just depends on how much you guys hate jenkins.
|
|
#
?
Dec 19, 2018 05:01
|
|
|
In other news, I accidentally a kubernetes today. We're running k8s in aws but have been raw dogging it with docker and shell scripts in our deprecated dev data center, finally hit a scaling problem with deploying more and more services as containers, and do a 1 to 1 service per dev environment. Turns out Rancher is rather good at setting up single host kubernetes machines. It's about 8 lines of shell and 10 mouse clicks to get it all set up. So that's one less deployment style I have to support.
|
|
#
?
Dec 19, 2018 06:33
|
|
|
Hadlock posted:In other news, I accidentally a kubernetes today. What are mouse clicks?
|
|
#
?
Dec 19, 2018 07:13
|
|
|
I have a customer that's using Azure. We built out a bunch of Azure stuff for them using ARM templates. They are now insisting that everything be redone in Terraform. I read up on Terraform and... I don't get it. It looks like it's exactly the same thing as ARM templates, but third-party and lacking support for some critical Azure services that was use heavily (such as App Service Environments). The syntax is nicer than ARM templates, I'll give it that. I have been unable to get a clear answer to the question, "What benefit is Terraform going to provide that ARM templates do not provide?" New Yorp New Yorp fucked around with this message at 15:35 on Dec 19, 2018 |
|
#
?
Dec 19, 2018 15:33
|
|
|
If they have stuff in other cloud providers it would make sense. Then you're at least using one toolchain to manage things, even if you can't directly reuse the code. They might also just have previous experience with it, or feel like it'll be easier to hire people with terraform knowledge? If not those reasons, then idk and it sounds dumb.
|
|
#
?
Dec 19, 2018 15:38
|
|
|
Docjowles posted:If they have stuff in other cloud providers it would make sense. Then you're at least using one toolchain to manage things, even if you can't directly reuse the code. They might also just have previous experience with it, or feel like it'll be easier to hire people with terraform knowledge? Sure, it makes sense in a multi-cloud environment. Which they're not, and they're unlikely to become. They have no internal Terraform knowledge (hell, they barely have any ARM template or Azure knowledge to begin with). And AFAIK hiring is not a motivating factor. I just wanted to be sure I'm not missing some critical awesome Terraform feature that is worth throwing out several months of work and starting from scratch.
|
|
#
?
Dec 19, 2018 15:42
|
|
|
I've extended native Terraform resources for Azure (I added the support for VM boot diagnostics) and it's really simple to wrap existing ARM resource APIs even if you don't know Go. That shouldn't be a blocker. (This doesn't impact whether or not this is a good idea, which it doesn't seem to be.)
|
|
#
?
Dec 19, 2018 16:05
|
|
|
New Yorp New Yorp posted:I just wanted to be sure I'm not missing some critical awesome Terraform feature that is worth throwing out several months of work and starting from scratch.
|
|
#
?
Dec 19, 2018 16:56
|
|
|
Erwin posted:Perhaps whatever you're deploying to Azure has Terraform providers? E.g. if you're creating a Kubernetes cluster and a Gitlab server and *checks docs* ...Iunno, RabbitMQ or something, you can configure them end-to-end with one Terraform run. I don't think that's worth redoing all of the existing work, though. I'd default to starting with Terraform for any new project instead of the cloud provider's DSL, but I don't think it would be my number one priority to rewrite if the code already existed. Nope on all of those points. 
|
|
#
?
Dec 19, 2018 17:12
|
|
|
Sounds like you get to bill them for several more months of (re)work, then 
|
|
#
?
Dec 19, 2018 19:05
|
|
|
The reason I’d advocate for using Terraform is that it’s easier to pick up for newcomers overall than trying to learn a cloud-specific DSL like CloudFormation. Hell, nothing stops me from deploying CloudFormation stacks as a Terraform resource anyway but I can’t do the reverse so the migration path is there.
|
|
#
?
Dec 19, 2018 21:10
|
|
|
Custom CloudFormation resource backed by a lambda that runs terraform 
|
|
#
?
Dec 19, 2018 22:06
|
|
|
Triggered ...and not by a lambda or SNS notification either
|
|
#
?
Dec 19, 2018 23:07
|
|
|
CloudFormation is a lot more attractive now with the release of the CDK.
|
|
#
?
Dec 20, 2018 00:52
|
|
|
necrobobsledder posted:The reason I’d advocate for using Terraform is that it’s easier to pick up for newcomers overall than trying to learn a cloud-specific DSL like CloudFormation. Hell, nothing stops me from deploying CloudFormation stacks as a Terraform resource anyway but I can’t do the reverse so the migration path is there. Right, the thing that should stop you though is that Terraform does not support a rich templating language of any sort which should result in the question, what the gently caress is this tool actually doing for me? Something that is reasonably interesting to me lately is actually reversing this, using Ansible to manage the composition and deployment of a terraform state through the new-ish terraform module. Unfortunately the module kind of sucks because it's still impossible to get terraform plan output in any kind of machine readable format (and it's not coming in 0.12 either). It's a comedy option tool -- something to embrace if your idea of infrastructure as code is chaining together shell commands, kind of like the aforementioned AWS CDK which turns CloudFormation into "npm install; cdk init app". Sidenote I understand the ansible guy is working on a new tool: https://github.com/opsmop/opsmop which is allegedly designed with the intention of being easier to attach to exist software. This is pretty interesting because one of my least favorite parts of ansible is that you really do need to run AWX to get the most out of it, and I hate running things just in general.
|
|
#
?
Dec 20, 2018 01:16
|
|
|
12 rats tied together posted:Right, the thing that should stop you though is that Terraform does not support a rich templating language of any sort which should result in the question, what the gently caress is this tool actually doing for me? The more PRs we get that are just "a set of files where someone just search+replaced the environment", the more I'm convinced our team is terrible. Motherfucker do you seriously just stamp out yamls all day? We have this problem with our k8s and Terraform definitions, and Terraform modules only go so far. Overall though I still like Terraform over our previous deployment tool, Spinnaker
|
|
#
?
Dec 20, 2018 01:58
|
|
|
terraform is certainly an imperfect and frequently buggy tool, but also the least bad one available for certain types of problems
|
|
#
?
Dec 20, 2018 02:09
|
|
|
Terraform's simplicity is its greatest strength. It wires together APIs from lots of providers in an idempotent and mostly ignorant way, and it doesn't try to reinvent any of their functions. Need config management? Use it and tie it into your VM lifecycle. Need a higher-level construct to create your Terraform graphs? Use your favorite language and output JSON instead of writing HCL. I understand why it's not a great fit for many use cases and workflows, but I'm happy that it does useful things instead of being a half-implemented mess of useless abstractions.
|
|
#
?
Dec 20, 2018 06:03
|
|
|
Vulture Culture posted:Terraform's simplicity is its greatest strength. It wires together APIs from lots of providers in an idempotent and mostly ignorant way, and it doesn't try to reinvent any of their functions. Need config management? Use it and tie it into your VM lifecycle. Need a higher-level construct to create your Terraform graphs? Use your favorite language and output JSON instead of writing HCL. I understand why it's not a great fit for many use cases and workflows, but I'm happy that it does useful things instead of being a half-implemented mess of useless abstractions. I still think the count thing is dumb.
|
|
#
?
Dec 20, 2018 06:08
|
|
|
Methanar posted:I still think the count thing is dumb. e: it looks like we get for/for_each, but they definitely still feel like declarative HCL: https://www.hashicorp.com/blog/hashicorp-terraform-0-12-preview-for-and-for-each Vulture Culture fucked around with this message at 06:19 on Dec 20, 2018 |
|
#
?
Dec 20, 2018 06:17
|
|
|
I don't think that I would ever allow a contractor to write a system using a proprietary tech when a comparable open source alternative exists, especially when terraform is sort of the lingua Franca in this space. Tools like kops have an option to export their commands as terraform. If it's in terraform, at least the next contractor they hire will probably have experience with terraform. With ARM the pool of contractors who can help them is quite small. Maybe a better example would be to say that your telling them that your preferred solution is to use Oracle, and they're telling you to use postgres instead, even though they aren't using any postgres specific features and Oracle has some cool functions that will make your life easier.
|
|
#
?
Dec 20, 2018 06:31
|
|
|
Smugworth posted:The more PRs we get that are just "a set of files where someone just search+replaced the environment" To give some context to my rather strong opinions about terraform: I have absolutely 0 tolerance for copy paste and change a thing. It's criminal that they went through all the trouble to write yet another DSL but you can't subclass a module. You can't even run interpolations in state configs or module paths last I checked which is an extremely galaxy brain approach to a declarative infracode tool. If something is different, override that property and surface it to me. Don't make me code review a +400 line diff where 394 of them are an exact copy paste. I have no patience for that kind of poo poo when inheritance as a concept has existed for longer than I've been alive.
|
|
#
?
Dec 20, 2018 07:40
|
|
|
Vanadium posted:Custom CloudFormation resource backed by a lambda that runs terraform Wonder if anybody has ever had an incident where one of their cloud resources had a startup trigger that spun up another 2+ resources and it Von Neumann'd up all their budget and/or the datacenter capacity in a few minutes.
|
|
#
?
Dec 20, 2018 09:09
|
|
|
Hadlock posted:I don't think that I would ever allow a contractor to write a system using a proprietary tech when a comparable open source alternative exists, especially when terraform is sort of the lingua Franca in this space. Tools like kops have an option to export their commands as terraform. If it's in terraform, at least the next contractor they hire will probably have experience with terraform. With ARM the pool of contractors who can help them is quite small. Maybe a better example would be to say that your telling them that your preferred solution is to use Oracle, and they're telling you to use postgres instead, even though they aren't using any postgres specific features and Oracle has some cool functions that will make your life easier. Scalr are going allow direct Terraform into their farm templates in second quarter next year which is pretty sweet. They came to the same conclusion really, why reinvent cloud integration when Terraform has done it already.
|
|
#
?
Dec 20, 2018 16:46
|
|
|
NihilCredo posted:Wonder if anybody has ever had an incident where one of their cloud resources had a startup trigger that spun up another 2+ resources and it Von Neumann'd up all their budget and/or the datacenter capacity in a few minutes. There are some horror stories with recursive lambda functions that go something like this. My previous job we also provided basically container scheduling as a service and the product was only up in internal preview for a couple of weeks before someone pushed a workflow with a slowly expanding cycle (some kind of model scoring job or something) and blew up production. The thing that gave out first was actually the VPC DNS server at the +2 address, where we blew past the 1k requests per second per eni limit from a caching resolver. We started investigating what looked like a DNS problem only to find that inbound requests had gone from a couple hundred per second up to the high 20,000s or something like that.
|
|
#
?
Dec 20, 2018 17:31
|
|
|
Vulture Culture posted:It's definitely weird syntax for expressing a loop. It's still not clear to me whether this will have an alternative syntax in HCL2/Terraform 0.12. Hi, I'm posting from the future from year 2321. Terraform 0.12 is still not released.
|
|
#
?
Dec 21, 2018 00:08
|
|
|
John Titor came back to warn us not to adopt Terraform 0.12 for production usage. He was trying to find a time period before companies started adopting 0.12.
|
|
#
?
Dec 21, 2018 00:30
|
|
|
Gyshall posted:Hi, I'm posting from the future from year 2321. Terraform 0.12 is still not released. Hi, I'm intensely demanding of an open source tool I don't pay for.
|
|
#
?
Dec 21, 2018 00:34
|
|
|
Not demanding. Just saying. We're also an Enterprise Terraform shop and tend to keep our templates as simple as possible.necrobobsledder posted:John Titor came back to warn us not to adopt Terraform 0.12 for production usage. He was trying to find a time period before companies started adopting 0.12. Love a good Titor reference.
|
|
#
?
Dec 21, 2018 03:23
|
|
|
12 rats tied together posted:Something that is reasonably interesting to me lately is actually reversing this, using Ansible to manage the composition and deployment of a terraform state through the new-ish terraform module. I spent some time today tinkering with this and my findings are: - Don't use it if you can avoid it - After spending 6 months away from terraform I never want to go back to that hell However, you do get for loops (anywhere in terraform, not just on the resource types that will support it in 0.12), nullable keys, the ability to run interpolations anywhere in a terraform config file such as state configuration and module sources, and with some effort you can just stop using modules entirely and just start declaring/inheriting/overriding resource definitions from some external source that don't need to exist as a resource inside the terraform dag. You can also link states together so you also don't need to write a makefile for applying from A and then B which is pretty nice. As expected it will be better if terraform ever supports json plan output, but until then its about as bad as using the CLI so it's pretty solid overall if your toolchain already includes both terraform and ansible.
|
|
#
?
Dec 27, 2018 04:21
|
|
|
Traditional DBA checking in. What is the current hot pattern for ephemeral database processes with long lived storage?
|
|
#
?
Jan 2, 2019 08:02
|
|
|
Open ended question. Ideally your app is stateless. Use something like flyaway or whatever to handle database updates and migrations. Use RDS or a similar DBaaS. Alternatively, NoSQL.
|
|
#
?
Jan 2, 2019 13:06
|
|
|
goatsestretchgoals posted:Traditional DBA checking in. What is the current hot pattern for ephemeral database processes with long lived storage? still presto, if i understand your question
|
|
#
?
Jan 2, 2019 15:39
|
|
|
IIUYC, the storage is permanent but the DB process itself is ephemeral (i.e. not a long-running daemon that clients connect to)? If that's the case, then SQLite works very well. The process running it can be terminated with almost a guarantee of no data loss or corruption. Postgres in a container would probably work well too; like SQLite it uses WAL files so it's fairly resilient to being killed at random, but I suspect (without evidence) that SQLite might be better.
|
|
#
?
Jan 2, 2019 19:26
|
|
|
Pile Of Garbage posted:Scalr are going allow direct Terraform into their farm templates in second quarter next year which is pretty sweet. They came to the same conclusion really, why reinvent cloud integration when Terraform has done it already. Azure integrates very well with Terraform now and it's good an insane plugin for vscode apparently (I use intellij personally so I can't vouch but some swear by it). Like others of mentioned the biggest win of Terraform is that you only have to learn the extremely simple (especially with an editor) language once and you have your automation/iaac/infra documentation solution half done regardless of what cloud you're deploying to. Other benefits are; - It interacts very well with Vault, so you can do things like store S3 keys straight into the password vault without having to ever see them - The state file is JSON so you can easily scrape it for information (within reason, remember there's secrets in there) - it plays remarkably nicely with existing resources, that is, you don't have to terraform your entire environment but you can still use it without fear of blowing poo poo away. I echo the sentiment that I would absolutely one hundred million percent not waste a second of time on any cloud specific deployment language.
|
|
#
?
Jan 3, 2019 03:42
|
|
|
|
| # ? May 15, 2024 04:33 |
|
|
Terraform doesn’t support some services in AWS at all (specifically, Data Pipeline) but I’m going to start using Terraform to deploy CloudFormation templates. But I’m about to write one (it is a complete pain in the rear end because it is the only AWS service I’ve seen that seems to have a non-deterministic provisioning state) and warn the world of the evils of Data Poopline.
|
|
#
?
Jan 3, 2019 04:55
|
|