|
necrobobsledder posted:Last I could tell from the outside, half the CloudFormation blogposts and engineers appeared to be based out of India which makes me wonder if it's strategically important enough for AWS to put more higher profile engineers on them. Doubtful you'd see that happen to IAM, in contrast. The CloudFormation team is based in Seattle and Vancouver. Are you basing this off names? New Yorp New Yorp posted:Containers / Docker Compose have nothing to do with Azure. Counterpoint: your customers don't use units, they use your application. Write unit tests so that you better define constraints between parts of your codebase, but unit testing actually shows how you what customers are experiencing. Make sure you make integration testing easy and consistent for your team so they write many of them.
|
# ? Apr 30, 2020 04:06 |
|
|
# ? May 15, 2024 21:43 |
|
Spring Heeled Jack posted:To anyone running k8s, how are you handling database migrations with deployments? We’re a .net shop and our pipelines (azure devops) are triggered to build a container for the migration and one for the service and run them in that order (the migration container runs as a job). our deployment tooling centers is a bunch of Python that consumes a YAML written in an internal schema and produces all our deployment artifacts. k8s YAMLs and then Datadog monitors and dashboards mainly. Sounds like overkill but we have like 8 teams with dozens of services between them, and getting them all up to speed on k8s would be a horrible pain. The schema dev uses is deliberately limited in some ways, so e.g. they get one Deployment per microservice and no funny poo poo with StatefulSets or God forbid creating pods directly. But then we've put a lot of effort into defining a rich schema for SLOs and providing tool support for more of the esoteric k8s features like HPA and PDB, so that it is easier in fact to go along with conventions and deploy your service with the provided tooling. I am a huge fan of the restricted schema for defining deployments. It helps a ton with reducing boilerplate and also with reducing variance between how different teams build and operate their services.
|
# ? Apr 30, 2020 04:18 |
|
FamDav posted:The CloudFormation team is based in Seattle and Vancouver. Are you basing this off names?
|
# ? Apr 30, 2020 04:37 |
|
Spring Heeled Jack posted:To anyone running k8s, how are you handling database migrations with deployments? We’re a .net shop and our pipelines (azure devops) are triggered to build a container for the migration and one for the service and run them in that order (the migration container runs as a job). We run jobs before rolling out the new version pods. The pods have initContainers also but they're more of a "is the database available and properly migrated" check. Keep in mind that those run on each pod, which is fine if your migrations NOOP if they see they've run aleady, but if they don't you'll probably want the job. Helm can be as good or as bad as you want it, but with the removal of Tiller I can't see much reason to hate on it. Sure, you can go full crazytown on the templating, and we do because we're building a chart as a vendor to support many different configurations, but it's perfectly happy to just render a static manifest as a "template" while still allowing you to use the various lifecycle hooks. Those you can use to automatically run the migration job and then only spawn the new version pods after it completes.
|
# ? Apr 30, 2020 04:46 |
|
Nomnom Cookie posted:[...] But then we've put a lot of effort into defining a rich schema for SLOs and providing tool support for more of the esoteric k8s features like HPA and PDB, so that it is easier in fact to go along with conventions and deploy your service with the provided tooling. It's kind of sad how autoscaling and disruption budgets suddenly become an esoteric feature when you move from, say, EC2 to kubernetes. Not a dig on your developers, mine also all read the same medium post that tells you to create StatefulSets for some loving reason, but in a lot of ways the shift to k8s can be a significant step backwards in functionality (assuming you were actually good at operating your previous provider(s)). From a product engineering perspective I haven't built anything on k8s in the past 3 years that isn't just a shittier version of AWS::AutoScaling::AutoScalingGroup. I think it's absolutely the right call to either build some kind of abstraction that your dev teams just feed structured inputs to, or to run embedded infrastructure engineers in every dev team. However, if you're speccing out a k8s abstraction, the AWS::ECS CloudFormation namespace is right there already, so IMHO it ends up being a huge wheel reinventing exercise. I also don't know if this is just a thing at my current employer or not, but every time someone brings up building an abstraction for something immediately like 10 people appear and start bikeshedding UI tech and web frameworks and poo poo, it's extremely tedious. CMYK BLYAT! posted:The pods have initContainers also but they're more of a "is the database available and properly migrated" check. Keep in mind that those run on each pod, which is fine if your migrations NOOP if they see they've run aleady, but if they don't you'll probably want the job. In particular here we have a task that maintains the state of a postgres db for airflow, which includes bootstrapping, ensuring migrations are properly ran, ensuring our additions to it (CI user, CI user permissions, etc) are in the correct state, and stuff like that. The same task actually creates the database in AWS too, which lets us create extra airflow environments automatically on merges to master. To me it seems like running database migrations should be part of the application deploy, and running an isolated stateful database in k8s and spinning up containers just to run some SQL feels unnecessarily complicated for a fairly solved problem, but OP did say .net shop, so
|
# ? Apr 30, 2020 06:50 |
|
Ami reading this right that you can't stand up Amazon Workspaces with terraform? Just the VPC and the Directory server?
|
# ? May 8, 2020 18:32 |
|
Matt Zerella posted:Ami reading this right that you can't stand up Amazon Workspaces with terraform? Just the VPC and the Directory server? Yeah, because a lack of API support from AWS: https://github.com/terraform-providers/terraform-provider-aws/issues/434 e: Oh, there's progress: https://github.com/terraform-providers/terraform-provider-aws/pull/11608
|
# ? May 8, 2020 20:46 |
|
But I can stand up the VPC for it? Or is that not a good idea if I'm plunking manually created stuff into it?
|
# ? May 8, 2020 21:14 |
|
Matt Zerella posted:But I can stand up the VPC for it? Or is that not a good idea if I'm plunking manually created stuff into it? The VPC resources you create in Terraform don't care about what you put in it, and it'd create a good base to import the other Workspaces resources once it's supported. I'd go for it.
|
# ? May 8, 2020 21:16 |
|
Bhodi posted:I'm stuck in this hell right now, trying to find the best way to build a multi-platform file using yum_repository from a yaml structure but apparently you can't have an inner and outer loop in ansible that traverses a tree with different size leaves without include_task? Because the jinja2 product filter only let you reduce not reference? Like, WTF? Vulture Culture fucked around with this message at 13:50 on May 9, 2020 |
# ? May 9, 2020 13:47 |
|
Vulture Culture posted:Is it difficult to generate this via Jinja templates rather than doing all the weirdo machinations in Ansible YAML?
|
# ? May 9, 2020 16:16 |
|
The next time you have to do something super wacky like this to generate multiple resources from bonzo logic, look into creating an action plugin. I've got a few good examples from modernizing a neglected repo in the past couple weeks
|
# ? May 9, 2020 17:05 |
|
Deferring to the group_vars plugin as often as possible is a great way to avoid scenarios like this (in particular, it's unlikely a system will ever be both rhel 7.1 and rhel 7.2 at the same time), but also in general, ansible-playbook loops are much easier if you keep your nested types the same type all the way down:code:
code:
e: Basically, however counter-intuitive it may seem, there's no reason to ever intentionally track variables in a list inside ansible. The only times you should do this are when you really need strict ordering plus some kind of index math tomfoolery, and hopefully those times never occur. 12 rats tied together fucked around with this message at 17:40 on May 9, 2020 |
# ? May 9, 2020 17:34 |
|
I'm trying to set up a basic CI/CD pipeline using GitLab CI/CD and I'm running into some weird errors. The project is just a static website served from an nginx docker container. I'm using gitlab.com, so this is using a shared docker runner. An additional wrinkle is that the VPS I'm deploying to is running centos8, so I'm building with docker and deploying with podman. Here's the complete .gitlab-ci.yml: code:
code:
code:
Any thoughts on what might be causing my issues?
|
# ? May 10, 2020 20:16 |
|
What is it called in Azure DevOps when committed code gets rejected if it doesn't build and pass all tests? People from our team break our pipelines all the time and I'm sick of it.
|
# ? May 11, 2020 07:47 |
|
Boz0r posted:What is it called in Azure DevOps when committed code gets rejected if it doesn't build and pass all tests? People from our team break our pipelines all the time and I'm sick of it. A good thing(tm).
|
# ? May 11, 2020 08:14 |
|
Mr Shiny Pants posted:A good thing(tm). Thank you, but I can't find out how to set it up if I don't know what they call it.
|
# ? May 11, 2020 08:52 |
|
Boz0r posted:What is it called in Azure DevOps when committed code gets rejected if it doesn't build and pass all tests? People from our team break our pipelines all the time and I'm sick of it. branch policies, i guess? https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops
|
# ? May 11, 2020 09:00 |
|
Asleep Style posted:Gitlab CI/CD deploy troubles In the end it turns out my problem was the bash -c in this line: code:
code:
|
# ? May 11, 2020 13:49 |
|
Boz0r posted:What is it called in Azure DevOps when committed code gets rejected if it doesn't build and pass all tests? People from our team break our pipelines all the time and I'm sick of it. You're describing something like a pre-push hook, which isn't really supported in Azure DevOps. What you're after are branch policies. A branch policy puts required conditions on completing a PR. With a branch policy in place, people can't push directly to a protected branch; they can only get things into that branch via a PR. So the workflow would become create branch -> work in branch -> push branch -> open PR -> builds/etc run -> PR can be completed once all policies are fulfilled.
|
# ? May 11, 2020 15:00 |
|
Are you using Azure DevOps for code storage as well or just pipelining and putting your somewhere else like GitHub? And to be more specific are build and tests failing on code that's supposed to be "production" ready? Because obviously you won't know for sure if it builds and passes tests until you've run it through a pipeline that will build it and and run tests, but you don't have to "commit to master" so to speak to get that.
|
# ? May 12, 2020 00:58 |
|
Docjowles posted:I agree trying to use terraform in a vendor neutral way is nuts at best and impossible at worst. But what they may have meant is that with a vendor specific tool, if you end up switching clouds (or jobs) your knowledge of the old tool is pretty much useless. Whereas once you know how to use terraform on Azure, you know how to use it on Amazon or GCP or a bunch of other things. You don’t have to relearn from zero even if you can’t directly reuse your code. And that has value. we’ve got Atlantis running on our development environment, it’s alright for keeping track of who did what where
|
# ? May 25, 2020 20:17 |
|
Anyone have strong opinions one way or another on Jenkins Job Builder vs. Job DSL?
|
# ? Jun 5, 2020 14:24 |
|
Jenkins Job Builder is really good if you need to manage hundreds or thousands of jobs with similar structures. DSL is better if you want to let teams manage their job structure. We use both. JJB to manage permissions, job structure, etc. Certain jobs point to Jenkinsfiles or whatever for teams that want stuff other than our standard build/test stuff.
|
# ? Jun 5, 2020 15:02 |
|
We use JJB (and have also written a JJB builder tool) and teams manage Jenkinsfiles. Our JJB configuration manages several thousand matrix jobs and across dozens of teams and several independent clusters. The documentation for Job DSL was not as thorough or as easy to search for compared to JJB a while ago so user friendliness won.
|
# ? Jun 5, 2020 16:28 |
|
We use job DSL for the precise reason Gyshall mentioned: it let's our teams manage things. Of course it really means that product teams complain to the platform team every time something goes wrong because why bother reading logs or trying to understand how systems work?
|
# ? Jun 5, 2020 17:08 |
|
We use dotnet pack to build nuget packages. How do I add a branch name to the package name?
|
# ? Jun 8, 2020 08:39 |
|
Boz0r posted:We use dotnet pack to build nuget packages. How do I add a branch name to the package name? I would strongly recommend putting the branch name in the version tag instead (eg "1.2.3.4-bozorsexperiment"), or in some other annotation. It's a fairly conventional assumption that project name == package name == root namespace, and while I don't think you would actually break stuff by doing things differently, it's highly unintuitive. The only reason I can think of to put the branch name in the package name is so that you can add two different branches as dependencies at the same time, which I dearly hope is not necessary, as it could easily lead to a mess of old unmaintained branches sticking around because other projects depend on them.
|
# ? Jun 8, 2020 09:38 |
|
NihilCredo posted:I would strongly recommend putting the branch name in the version tag instead (eg "1.2.3.4-bozorsexperiment"), or in some other annotation. It's a fairly conventional assumption that project name == package name == root namespace, and while I don't think you would actually break stuff by doing things differently, it's highly unintuitive. Yeah, that's what I want to do. I forgot that difference between name and version tag EDIT: I've tried setting an environment variable, but I get an error: code:
Boz0r fucked around with this message at 10:17 on Jun 8, 2020 |
# ? Jun 8, 2020 09:41 |
|
Boz0r posted:EDIT: I've tried setting an environment variable, but I get an error: How/where are you trying to do this? What process are you using? Where are those variables supposed to be coming from?
|
# ? Jun 8, 2020 16:19 |
|
My new gig has a requirement that we have to run portions of the app on premise in a data center. I'd like to handle everything with APIs and/or Ansible if possible. I've been out of the hardware game for a bit, but my thought was to do some amount of hyper converged infra across two racks. What's the best stuff these days for handling PXE? Last I did anything was MaaS. How about firewalls/switches? We want to avoid Cisco or any stuff like that. Kind of out of the loop on that as well.
|
# ? Jun 10, 2020 02:39 |
|
For PXE last I ever did anything that worked reliably was running legacy stacks in IaaS lift and shift configurations after MaaS didn't work out for us, so I'm kind of at a loss there. If you're not going to be doing hardware switching, you should be prepared to lose a bit of latency (hardware is still superior to software outside of what hyperscalers can justify). Juniper has vMX that is one of the better software routers in the SDN landscape at least for enterprisey folks. And honestly, if you're into SDN management, you should be looking to work with NAPALM supporting vendors and use either Ansible or Saltstack for management.
|
# ? Jun 10, 2020 04:35 |
|
State of the art is something tinkerbell.org or digital rebar, which use iPXE to drive advanced workflows.
|
# ? Jun 10, 2020 04:45 |
|
Yeah I started playing with Digital Rebar last night. Seems like that will handle the metal scaling pretty well. I've used MaaS before but I think we'll need something a bit more robust. I'll take a look at the other stuff. Any good reference architecture out there for these kind of setups?
|
# ? Jun 10, 2020 13:18 |
|
My employer uses cumulus linux devices as TOR switches and I'm definitely not qualified to offer you an objective evaluation of them but I understand that they are cheap, and I like working with them a lot because they are just linux boxes. PXE we're using foreman, which seems to kind of suck, but it seems like everything sucks down there in PXE land.
|
# ? Jun 10, 2020 15:23 |
|
Are projects like maas and tinkerbell useful in more traditional provisioning environments? Where I'm at we're typically 5+ years behind the cutting edge, so we're not very cloud aware though the momentum is building. Now that everyone is work from home provisioning new hardware has gotten really difficult because our workflow is fairly antique.. get some new hardware, someone spends a day configuring IPMI and customizing a ks.cfg then letting pxeboot do its thing. This hasn't evolved much for almost 20 years even though we've iterated through stuff like cobbler and foreman. So if there's some better provisioning tools out there that support traditional servers (that is, it's provisioned and performs a single task until the hardware fails), that is pretty interesting to me. There has been talk of moving user analysis jobs into openshift which I'm not sure I'm keen on but it does indicate a more dynamic solution is on the horizon.
|
# ? Jun 10, 2020 15:59 |
|
Is this the right thread for what's likely a super basic docker question? I'm trying to run graphite and grafana on the same VM. If I start the graphite container before the grafana container everything works fine. If I start the grafana container first the graphite container doesn't operate correctly. If I shell into the container everything looks fine - at least as far as I can tell - I'm not a graphite expert. The log files all look clean and I'm able to access the web based dashboard via wget and localhost. Externally I'm not able to talk to the graphite container at all over 80 and the grafana container is unable to connect to the database over 8080. Given that these both run through different processes ( port 80 -> nginx and port 8080 -> gunicorn ) it seems like something basic is busted. The 2 containers don't share any ports - here's what it looks like when things are running properly code:
|
# ? Jun 10, 2020 17:22 |
|
^ use Prometheus, unless you absolutely have to use graphite. It works out of the box and it's the new standard for that sort of thing So my question, Our analytics team is working from a backup/restore of the proof database that happens nightly. Now they want near-real-time data. This analytics db is walled off from regular users and approved by it security etc. It's a postgres 9.6 db that's going to 11.x soonish I was thinking about still doing the dump/restore on say, Sunday nights, and then restore from WAL files using Wal-E to manage write-ahead logs and setup the analytics db as a delayed replica The advantage of this is I'm not consuming a replication slot, I'm not consuming database connections etc as I'm keeping the DB up to date via WAL files stored on S3 We used Wal-E at a previous company to do something similar, I think they're still using it to this day I'm curious if there's a better way to do this. A quick Google shows that Heroku and Gitlab are doing exactly this so I don't think I'm breaking any new ground
|
# ? Jun 10, 2020 21:08 |
|
fankey posted:Is this the right thread for what's likely a super basic docker question? I'm trying to run graphite and grafana on the same VM. If I start the graphite container before the grafana container everything works fine. If I start the grafana container first the graphite container doesn't operate correctly.
|
# ? Jun 11, 2020 00:52 |
|
|
# ? May 15, 2024 21:43 |
|
If you need a quick-fix use docker-compose and include a depends_on statement to control start-up order. Still would be better to work-out what the issue is ofc
|
# ? Jun 11, 2020 11:44 |