|
What are people using to manage/implement secrets in k8s? The built in method of "creating the secret > mounting the secret as a volume in the pod > defining all secret values you will need in the mount > referencing the secrets in env values" seems like a huge goddamn hassle for a pod that's going to have like 10+ secret env variables.
|
#
?
Feb 21, 2019 14:42
|
|
|
|
| # ? May 15, 2024 07:30 |
|
|
terraform 0.12 is going to include a lot of features that are lacking in terraform now. It's going to have stronger types, loops, not require everything to be string interpolation, basically make terraform HCL look more like code and not scripts that just assemble strings
|
|
#
?
Feb 21, 2019 14:44
|
|
|
Cancelbot posted:I don't get the Terraform (mild?) hate - We have 24 AWS accounts with over 200 servers managed entirely in Terraform. Recently our on-premises subnets changed and it took all of 10 minutes to roll it out to everyone's code. In my case, it's because the Azure provider sucks and doesn't work very well for the scenarios where my client wants to use it. The Terraform fans I've discussed it with have a surprising tendency to blame the Azure platform instead of accepting that the tooling itself may be lacking or poorly implemented.
|
|
#
?
Feb 21, 2019 16:16
|
|
|
New Yorp New Yorp posted:In my case, it's because the Azure provider sucks and doesn't work very well for the scenarios where my client wants to use it. The Terraform fans I've discussed it with have a surprising tendency to blame the Azure platform instead of accepting that the tooling itself may be lacking or poorly implemented.
|
|
#
?
Feb 21, 2019 16:19
|
|
|
Bhodi posted:Anyone have a good aws terraform example config for something like that? We're bringing up a new vpc from scratch and want to switch to autogenerating the subnets, sgs, iam roles and such and I don't really want to fall into any obvious traps. Disclaimer: we are still figuring out terraform so everyone please feel free to tell me this is wrong and dumb. But we already did it once, and had everything suck, and have now learned from those mistakes so are hopefully redoing it better. The community VPC module is pretty solid for stamping out VPCs quickly and uniformly. At lest the VPC itself and subnets and NAT gateways, all the networking pieces. IAM and SG's you're on your own. The most important trap to avoid is putting all your poo poo into one state file. Just as you said, you don't want to be updating the tag on a dev instance and somehow accidentally delete your whole VPC. TF will only consider .tf files in the current directory as in scope for changes. So if you dump ALL of your code in one directory, the whole thing is going to get evaluated and applied on every run. Instead, break your state up into logical units. We have our git repo broken down like code:Charity Majors has an excellent rant on the subject. Always do a plan, save it to a file, and then apply that plan. That way you're (more likely to be) just applying the changes you really indented to make, and can fully review what it's going to do beforehand. There are also some nice wrapper tools like Terragrunt that can cut down on the boilerplate poo poo you need to recreate in every directory.
|
|
#
?
Feb 21, 2019 16:37
|
|
|
Terraform provisioners and other kinds of hooks-as-resources mechanics are what you should be using instead of custom resources in CloudFormation if at all possible. I wrote a 4 line AWS club shell script that turned into a monstrosity of a lambda function because a resource created within CF needed to be peered with another two VPCs. I recommend no more than one VPC created within a stack as a result of the interlinking mess. Terraform really shouldn’t be used for application deployments though. Blue green deploys and rolling deploys are actually supported in CloudFormation at least, not some “you get to build this crap again with shell scripts!” solution like in Terraform.
|
|
#
?
Feb 21, 2019 17:23
|
|
|
Vulture Culture posted:Having written (tiny) parts of the Azure provider, it can definitely be both. Oh yeah, I wasn't claiming otherwise! I've just gotten a lot of weirdly defensive "well it can't be a problem with Terraform, it must be Azure's fault" when discussing issues that I've ONLY experienced while using Terraform, and not the portal, ARM templates, Azure PowerShell, or the Azure CLI.
|
|
#
?
Feb 21, 2019 20:49
|
|
|
The NPC posted:Is the answerfile included in the package? Can you use Resolve-Path on ./tools/foo.rsp to get the absolute path and pass that in? If you're passing it in through $silentArgs you might have to watch how quotes and variables get escaped too. Mother fucker I hate Oracle. So I think I found out what was the problem: Oracle uses a universal installer to do its, well, installs. Thing is, said installer isn't the first stop on the journey; it's the 12c setup executable in this case (Program A). Chocolatey only knows and cares about Program A and doesn't have the slightest worry in the world about what's going on with the universal installer (Program B). Near as I can tell, A calls B and passes it some info and closes out with a nonstandard exit code, because Oracle. Chocolatey sees the non 0 exit code, assumes everything's FUBAR and starts purging the install files that B needs so things get cleaned up. This causes B to error out randomly depending on what got deleted first. This doesn't happen when doing things "normally" with the same flags via a Powershell window because there's nothing there to really care. Resolution: Pass a flag to have A hang around until B finishes, resulting in an acceptable 0 exit code. Possible Alternative Resolution: Have 259 as an acceptable code for this one package and ignore the hang around flag. I'm starting to understand why everyone I've ever worked with that's been in Ops beyond a certain year range sounds like they hate their jobs/lives.
|
|
#
?
Feb 21, 2019 22:30
|
|
|
freeasinbeer posted:Small tasks become cumbersome, like making a small tweak is very easy to do in the UI but maybe hard and time consuming in tf. And if you do use tf since it can reconcile global state, it sometimes starts editing things you don’t want. Peering for example is easier to do in the UI across accounts, but then adding those routes in is a nightmare. We give each team an account and they look after their own resources (fully autonomous woo!). We set hard borders where they can only talk with HTTPS cross accounts unless its old legacy crap like SQL in which case we use PrivateLink/VPC peering only where necessary; this saves on shared state and we don't have to gently caress about with routing or IP clashes unless they need to hit the stupidly monolithic SQL cluster or get to our on-premises infrastructure. Oddly enough for the scale we are I feel as though we are light/efficient on server resource; Our website runs on 4x m4.2xlarge servers as CloudFlare sucks most of it up and our back-end hasn't been fully moved to AWS yet, which is another 200 servers in waiting. We also have some bad habits such as 30 services on a single node which is why we're pushing hard on the ECS end of things. For Terraform & containers we are having some fun; we did once define tasks & services with Terraform, but the developers love Octopus deploy too drat much and it was breaking all sorts of things. So instead we made enough Octopus plugins for it to play nice with ECS at a level the devs like; "Deploy my app", "Open a port", "Scale up" etc. all Terraform does there is build the cluster using a module. The idea behind us Terraforming everywhere is that these teams can then own their full stack in as faithful way as we can, and they can help each other out by looking at code instead of trawling through the inconsistent AWS console experience. Cancelbot fucked around with this message at 15:17 on Feb 22, 2019 |
|
#
?
Feb 22, 2019 15:14
|
|
|
Today in Kubernetes adventures: the mysterious Service that doesn't want to use one of its ports. ELB for the LoadBalancer comes up fine and maps the external ports to the correct nodePort. Traffic arrives successfully at nodes destined for the correct nodePort, but when directed into the Pod container itself, somehow both end up going to the same targetPort, apparently ignoring what's in the actual Service definition. This happens even that port/targetPort combination is flat out removed from the Service, which makes no sense. Either there's actually some bug in that part of the network provisioning code, or I and a bunch of other people are missing something really dumb in the Service/Deployment definitions.
|
|
#
?
Feb 23, 2019 05:36
|
|
|
Crossposting from the AWS thread since I know there's a bunch of terraform geeks in hereDocjowles posted:Ok I have my own Route53 question. We were hoping to switch from managing our own internal resolvers to using route53. We created a new private hosted zone with like 1000 records in it using Terraform. It took ages to complete which I kind of expected. But it also takes ages to do a subsequent plan/apply even if there are no changes. Like 15 minutes per no-op run. Which is uh not going to fly for a frequently changing zone.
|
|
#
?
Feb 26, 2019 20:18
|
|
|
Docjowles posted:Crossposting from the AWS thread since I know there's a bunch of terraform geeks in here I'm curious what it's spending most of the time doing, are you able to tell from the output? If not, maybe need to turn up the logging level.
|
|
#
?
Feb 26, 2019 23:36
|
|
|
fletcher posted:I'm curious what it's spending most of the time doing, are you able to tell from the output? If not, maybe need to turn up the logging level. It'll be the "refreshing state" part of the plan. I think Terraform just has a list of "this R53 record should exist here" in its state file, which then fires off a metric poo poo-ton of AWS API calls to verify that is indeed the case. It'll then do a diff based on what is consistent with the AWS state & the new computed state, rather than be smarter by looking at the HCL that changed prior to doing the refresh.
|
|
#
?
Feb 26, 2019 23:56
|
|
|
Cancelbot posted:It'll be the "refreshing state" part of the plan. I think Terraform just has a list of "this R53 record should exist here" in its state file, which then fires off a metric poo poo-ton of AWS API calls to verify that is indeed the case. It'll then do a diff based on what is consistent with the AWS state & the new computed state, rather than be smarter by looking at the HCL that changed prior to doing the refresh. Regardless of whether the configuration actually changed, it would need to refresh every resource anyway, since it would need to modify any drift.
|
|
#
?
Feb 27, 2019 02:21
|
|
|
Yeah, it’s that. It spends 15 minutes refreshing the state. And we haven’t even imported all the zones we would have in production yet, lol, this is just a subset for a test. Probably going to end up writing our own tool to do this which isn’t terribly hard. I just always prefer to use popular off the shelf stuff first if possible. I was wondering if there was some obvious workaround or something since I assume we are not the first team wanting to manage large zones via terraform. But maybe I am uniquely dumb 
|
|
#
?
Feb 27, 2019 02:32
|
|
|
You don't have to have all the records in a single terraform config.
|
|
#
?
Feb 27, 2019 02:52
|
|
|
That terraform blog post was amazing, I need more words like that.
|
|
#
?
Feb 27, 2019 02:59
|
|
|
The Route53 API is also pretty hostile if you're not batching requests, and that's something that Terraform's architecture makes really hard. If you needed to hack around this, instead of route53_record resources, you could consider using local-exec provisioners to call out to awscli or something to keep it out of the state graph. This will work fine for resource creation, but will be murderously slow on deletes if you have a zone with a lot of records.
|
|
#
?
Feb 27, 2019 04:20
|
|
|
Spring Heeled Jack posted:What are people using to manage/implement secrets in k8s? The built in method of "creating the secret > mounting the secret as a volume in the pod > defining all secret values you will need in the mount > referencing the secrets in env values" seems like a huge goddamn hassle for a pod that's going to have like 10+ secret env variables. I implemented this in vault, but we're not operationally mature enough to auto roll vault keys, so that got scrapped for helm secrets that is backed by aws key value store and we can lean heavily on IAM instead, but also supports other back ends so we're not fully tied to Amazon. Nice thing about helm secrets is that it pulls any orchestration out of the container and now it's just depending on those values passed in as env vars, which is a lot more pure 12 factor Downside to helm secrets is that i haven't gotten it to cleanly helm lint/helm secrets lint yet, but that's more of a time/priority problem than a technical hurdle Edit: also the other big win was to put as many service dependencies into containers as possible, database, message queue, etc, and then put everything in it's own namespace. That allows you to hard code a lot of things like user names and service dns names (our primary db used to be primarydb-dev1.example.com) but now it's simply "primarydb" and pointing the app at primarydb:5432/primarydb, inside the namespace, resolves to the correct database. Significantly reduces the amount of config, since every namespace can have it's own service with the identical name "primarydb"... We went from 20+ config items down to about 6 for our most complex app (the monolith) Hadlock fucked around with this message at 10:03 on Feb 27, 2019 |
|
#
?
Feb 27, 2019 09:57
|
|
|
how do y'all do your container registry on gcp? meaning, if you have N environments, each inside their own gcp project... do you have each separate environment trigger off your repo and build their own containers that they put in their own registries in parallel/on-their-own? or do you create like a designated-registry-project that they all just pull from? if they share a registry-project then where are you putting the deploy hook that runs the 'kubectl set image' after the push? StabbinHobo fucked around with this message at 18:15 on Feb 27, 2019 |
|
#
?
Feb 27, 2019 17:54
|
|
|
StabbinHobo posted:how do y'all do your container registry on gcp? We use one great big container repo, each branch has a jenkinsfile that builds a namespace/monolith-branchname:build-id and then fires off whatever webhooks are relevant to that branch/jenkinsfile and/or a helm upgrade -n mydeploy --set monolithVersion="namespace/monolith-branchname:build-id"
|
|
#
?
Feb 28, 2019 02:16
|
|
|
My company has automated a PeopleSoft linux deployment in AWS using mostly Cloudformation, CodeDeploy and Ansible playbooks. We now have to deploy PeopleSoft on Windows and we're running into some issues with Ansible, mainly that Ansible can't be run on a windows box without a linux control box. I've seen that there are some work arounds like installing Linux Sub System, or cygwin, but I don't know how nice those will work with our current linux playbooks. I'm just curious if anyone has run into a similar problem and either used Ansible or another orchestration tool like Terraform. Any insight would be appreciated.
|
|
#
?
Mar 5, 2019 04:08
|
|
|
https://www.macrometa.co/blog/why-global-edge-fabric This is one of those posts where you’re about to quote but then your eyes scan down and you find something dumber. That said, my current favorite is quote:What we need is layered model that decouples its various layers - data storage, data organization, data manipulation and data model.
|
|
#
?
Mar 5, 2019 07:19
|
|
|
SnatchRabbit posted:My company has automated a PeopleSoft linux deployment in AWS using mostly Cloudformation, CodeDeploy and Ansible playbooks. We now have to deploy PeopleSoft on Windows and we're running into some issues with Ansible, mainly that Ansible can't be run on a windows box without a linux control box. I've seen that there are some work arounds like installing Linux Sub System, or cygwin, but I don't know how nice those will work with our current linux playbooks. I'm just curious if anyone has run into a similar problem and either used Ansible or another orchestration tool like Terraform. Any insight would be appreciated. Ansible runs fine under wsl or even docker
|
|
#
?
Mar 5, 2019 07:36
|
|
|
I spun up loki on our test kubernetes cluster and now I don't need to maintain an elastic search cluster just for logging  Since everything is deployed using a helm chart everything is already correctly tagged and sortable by custom labels, container name, namespace, etc You can't do full text search on it, but whatever, this serves 99.99% of our developer logging needs I setup Loki to have 120gb disk to start; we're using 500gb for Prometheus and that seems to be gross overkill, we're going to have 50% disk left over after the first year.
|
|
#
?
Mar 5, 2019 08:14
|
|
|
goatsestretchgoals posted:https://www.macrometa.co/blog/why-global-edge-fabric Holy crap that article is such total bullshit.
|
|
#
?
Mar 5, 2019 08:39
|
|
|
goatsestretchgoals posted:https://www.macrometa.co/blog/why-global-edge-fabric from a product perspective this is probably not where i would start building an ambitious edge-compute database product, but more power to them i guess Vulture Culture fucked around with this message at 15:27 on Mar 5, 2019 |
|
#
?
Mar 5, 2019 15:23
|
|
|
fluppet posted:Ansible runs fine under wsl or even docker Would the application we install with WSL ansible on the Windows box also run in the WSL or can I install it natively in Windows?
|
|
#
?
Mar 5, 2019 21:42
|
|
|
Messing around in my lab and trying to figure out a reliable way to deploy a node app to Windows. No containers, just a Window Server VM with nothing installed. I'm using Azure Devops Pipelines, and deploying the code from repo isn't a problem, I'm just not sure how to reliably ensure Node is present and if it is already running, how to reload the app.
|
|
#
?
Mar 8, 2019 22:57
|
|
|
The Fool posted:Messing around in my lab and trying to figure out a reliable way to deploy a node app to Windows. Do you really need it to be windows server? You're paying a pretty non-trivial amount in licensing costs to be running stuff on windows if you don't need to.
|
|
#
?
Mar 8, 2019 23:02
|
|
|
Methanar posted:Do you really need it to be windows server? I'm attempting to combine a hobby (the node app) with work (Windows Server Admin). If I was doing this 100% for myself I would just build a container on a linux vm and rebuild when I needed to deploy an update.
|
|
#
?
Mar 8, 2019 23:08
|
|
|
The Fool posted:Messing around in my lab and trying to figure out a reliable way to deploy a node app to Windows. Put a devops agent on it powershell release task - Install chocolatey - Choco install nodejs npm/yarn/whatever to deploy packages or distribute files from build artifacts via artifact download task
|
|
#
?
Mar 8, 2019 23:17
|
|
|
You can use pm2 to manage the node process and it looks like it has a windows service package to support it. So all you need is to get nodejs and npm on a machine and then install everything via npm
|
|
#
?
Mar 9, 2019 14:09
|
|
|
I work for basically an MSP that provides basically IaaS to our (internal) customers so devops is kind of tricky for us because there's no dev, only ops. Nonetheless we've automated a ton of stuff with powershell and Azure runbooks, and have written and opensourced a number of Powershell modules that we use all over our code. We ended up hacking together the very barest form of a build "pipeline" before any of us knew what that even meant. Everything is in Github (internal code in our on prem enterprise Github, open sourced in public Github) with everything hooked into an Azure runbook via webhook, and whenever stuff gets updated it gets deployed: either copied to a set of servers to run or copied automatically into the Azure runbook. Well I've been aware of CI for a little bit now and finally found enough info about building a pipeline for Powershell modules and I spent all day Friday on it and I've now got one of our modules plugged into both Appveyor and Azure Pipelines (used a tutorial that used Appveyor but then found Azure and got that working as well). It doesn't deploy yet but it does run some pretty basic Pester tests to ensure the code is valid Powershell and that it passes PSScriptAnalyzer tests. So now I'm one of you I guess.
|
|
#
?
Mar 10, 2019 05:10
|
|
|
FISHMANPET posted:pipeline for Powershell modules Nice, sounds pretty cool. Where are the tests being executed, inside containers or some serverless dealio?
|
|
#
?
Mar 10, 2019 12:57
|
|
|
Cancelbot posted:I don't get the Terraform (mild?) hate - We have 24 AWS accounts with over 200 servers managed entirely in Terraform. Recently our on-premises subnets changed and it took all of 10 minutes to roll it out to everyone's code. There are some pitfalls you can run into because either Hashicorp hewed too strictly to bring declarative (iterating over resources sucks, reminds me of early puppet), the language wasn't created very well (doesn't support nested maps, I think this is fixed in 0.12), or there's just pervasive annoying bugs (such as weirdness with variables in nested modules). For toy setups or rubberstamped environments, Terraform seems perfect. It just starts to suck when you try to add complexity in certain places. One group at my company wrote something to generate Terraform. This gets around some of the pitfalls.
|
|
#
?
Mar 10, 2019 16:28
|
|
|
Pile Of Garbage posted:Nice, sounds pretty cool. Where are the tests being executed, inside containers or some serverless dealio? I... don't know? They execute in whatever Appveyor or Azure Pipelines runs in. This is the module I've been working with: https://github.com/umn-microsoft-automation/UMN-SCOM/tree/build-pipeline My yaml files for both Appveyor and Azure are the same, they specify a Windows server image and then call build.ps1 in the Build directory, which calls psake.ps1 which runs the tests.
|
|
#
?
Mar 11, 2019 02:33
|
|
|
Let's say I will have some Linux machines that will be deployed around the world with intermittent connectivity ( think satellites ) running docker on some Linux variant. What would be a good way to keep these under control and up to date? Would it be possible to just run regular docker on them and push updates using something like Ansible?
|
|
#
?
Mar 11, 2019 17:42
|
|
|
I think it comes down to splitting the update into 2 phases: 1) Ensure the desired state description and necessary assets are on the machine. 2) Initiate a server-local process to update the desired state from those local assets. 1 can be retried until successful with no side effects, which mitigates the issue of poor or intermittent connectivity. And 2 is likely to succeed because all the assets are in place and no connectivity is required.
|
|
#
?
Mar 11, 2019 18:07
|
|
|
|
| # ? May 15, 2024 07:30 |
|
|
minato posted:I think it comes down to splitting the update into 2 phases: So something like: rsync the required files or updates and update from the local files? Is this something I need to build myself or are there some nice utilities available, I can't be the only one dealing with something like this.
|
|
#
?
Mar 11, 2019 19:00
|
|