|
Virigoth posted:Has anyone here used Spinnaker and has thoughts on it? Our system is a Jenkins master right now that supports 3 lower environments for dev, QA, and our staging environment and then 4 production regions in AWS. We manage around 2,000 servers for micro-services but are growing that number daily it seems. Our Jenkins master does about 16,000 jobs a day and uses a in-house jslave script to make sure we have enough executors in each environment. With the permissions needed to deploy this is a scary scenario. What I'm considering doing a proof of concept on is hiding Jenkins behind Spinnaker and using it just like the demo video sets up with Spinnaker being our front end and a lot of user access removed. Make sure you post how it goes. We're evaluating Spinnaker but we're still not entirely sure where to drop it into our pipeline and how to use it to replace all the stuff we use in Asgard.
|
#
¿
Mar 19, 2016 20:14
|
|
|
|
| # ¿ May 2, 2024 04:19 |
|
|
Cancelbot posted:bake AMIs This is the right way to do it. In the same way you consider a compiled binary an artifact of a given release, you should think of a baked AMI as another artifact of the same release. If you feel like you're wasting cycles using 1 instance per service you should either rethink your architecture, use smaller instance types, or decrease the size of your fleet as a whole. Alternatively, you can bundle multiple services into one baked AMI although it kind of breaks the paradigm and can be a concern if you're trying to run stateless apps.
|
|
#
¿
Oct 18, 2016 14:25
|
|
|
Your other option is using Azure for hosted services and not worrying about instances at all.
|
|
#
¿
Oct 18, 2016 19:57
|
|
|
Dreadrush posted:Hi I'm very new to the whole Docker thing and am trying to learn more about it. Don't deploy nginx for static files unless you can really help it. You'll find it requires more work to maintain than you want and you'll have to do some work to scale it for traffic even if it is as simple as putting containers behind Mezos or Kubernetes. Instead, use something like S3 or CloudFront (or w/e your cloud provider has) for static stuff and set CORS accordingly.
|
|
#
¿
Nov 20, 2016 19:05
|
|
|
EssOEss posted:Other than that, I have not encountered any issues. To those saying don't do it - why? A better question is why bother? Containers are designed to be ephemeral and short lived. That's why their deployment mechanisms emphasize scale up/scale down behavior and time-to-new-deployment speed. Deploying a db in a container in production just feels unnecessary at best and a data loss risk st worst.
|
|
#
¿
Jun 10, 2017 01:49
|
|
|
Mr. Crow posted:There are more reasons to use containers than them being ephemeral and there is nothing anywhere implying they need to be short lived. Do tell? I've only seen docker used in the context or easy deploys and hot swapping because of how fast they spin up and their isolation by design.
|
|
#
¿
Jun 12, 2017 21:44
|
|
|
necrobobsledder posted:bandwidth contracts (yes, they do have them to help lower egress costs substantially, mostly of use when you get to petabytes / mo in transfers). How does this work?
|
|
#
¿
Jul 3, 2017 01:56
|
|
|
Anyone dug into Spinnaker's guts? It's a horror show of "I want this for my cloud" with the vaguest sense of cohesion.
|
|
#
¿
Jul 27, 2017 23:10
|
|
|
Keep in mind that there are some major issues with scaling CloudFormation past a certain point. If you're already getting started with Terraform you may as well just do things the Terraform way and avoid those problems altogether. Also, don't do scaling groups with Terraform. Consider it this way, Terraform is how you set up your immutable-ish infrastructure; it's where your IAM roles, security groups, load balancers, S3 buckets, and the like go. Use something like Spinnaker to manage clustered services and their configuration instead.
|
|
#
¿
Jul 30, 2017 12:37
|
|
|
necrobobsledder posted:Postgres as service discovery, wtf. I'm about to use this https://github.com/adrianlzt/hiera-postgres-backend  my brother have you heard of zookeeper, etcd, or consul?
|
|
#
¿
Jul 30, 2017 16:59
|
|
|
goondolances
|
|
#
¿
Jul 30, 2017 19:53
|
|
|
necrobobsledder posted:Well yeah, Puppet and Chef are mostly useful for very stateful systems that shouldn't have nodes go up and down frequently and they're real awkward for elastic systems. I had enough problems with this with Chef node registration and de-registration. Fwiw we bake images and provision them using chef and then run chef-solo on every instance in the fleet to complete provisioning and do dynamic user management. We've been looking at converting part of our fleet to use chef server because we don't have a great way to provision parts of our fleet in different ways but I'm sure they'll bring a whole bunch of other issues.
|
|
#
¿
Jul 30, 2017 23:05
|
|
|
We've been using Archaius and a distributed sidecar service we wrote for configuration management and it works extremely well. We have an ohai plugin that we wrote with the intent of using it to tune different settings on running nodes but we've never actually used it in production. I'm hesitant to post the repo even though it's open source, though, for fear of doxxing myself.
|
|
#
¿
Jul 31, 2017 14:37
|
|
|
I'm glad SRE is a known thing because even though that's not my actual official title I can still point to it and say that's what I do rather than hand wave "devops-y engineering work"
|
|
#
¿
Aug 5, 2017 13:51
|
|
|
Pollyanna posted:Ive always seen SRE stuff described as "be on Pagerduty and get frantic calls when alarms go off" while "doing devops" is Docker, CICD, and AWS. SRE work is a superset of software engineering that deals not only with CI/CD but also platform stability. It can definitely have PagerDuty work but a lot of it is looking at existing architecture and performance and helping development teams identify bottlenecks and issue areas and helping them reengineer with an eye for operational improvement.
|
|
#
¿
Aug 5, 2017 23:43
|
|
|
OWLS! posted:Anybody going to Boston DevOps Days? I'm on my way there right now if only the T would run a little faster.
|
|
#
¿
Sep 18, 2017 13:49
|
|
|
Docjowles posted:as a Boston resident, I have bad news for you~ No trust me I'm intimately familiar with how bad the T is. I just usually avoid the orange line but welp
|
|
#
¿
Sep 18, 2017 14:21
|
|
|
Yeah, don't use Datadog for tracing (although I think they have an APM feature now) or event correlation unless it's very broad. That's what an ELK stack or Splunk is for.
|
|
#
¿
Nov 12, 2017 19:59
|
|
|
Pollyanna posted:Pay at startups is directly proportional to what they can raise from investors, VCs, and stakeholders, so buzzword bingo and hype plays a massive role. I don’t know much about pay at established companies and large corporations, but it tends to be much more tempered in reality. this is the exact opposite of reality unless you include equity which may not be worth anything as pay at startups
|
|
#
¿
Dec 11, 2017 16:10
|
|
|
Mr. Crow posted:Ok so you are being argumentative over a one-off anecdote of apparently dubious quality and then arriving at the same conclusion. A way to use it without ssh access to the node?
|
|
#
¿
Dec 14, 2017 13:49
|
|
|
You're hosed unless you have a massively forward-thinking ops team who will manage your cluster(s).
|
|
#
¿
Jan 24, 2018 13:10
|
|
|
Punkbob posted:deploying on kops on AWS in a week if you know what you are doing Kops has a lot of weird edge cases that are show-stoppers when they crop up like slotting into pre-existing infrastructure-as-code or using pre-existing bastion hosts. Also it's not CI friendly in any way. It made me sad 'cause the dev team is super nice and helpful, they just built it to fit their use case and then had to do a bunch of work to make it more generalized.
|
|
#
¿
Jan 24, 2018 22:57
|
|
|
I'm genuinely curious what sort of scale folks are running k8s at. Especially those who have been talking about migrating to it or quickly spinning up a new cluster.
|
|
#
¿
Jan 29, 2018 22:36
|
|
|
I was specifically referring to people posting in this thread but talks about automating k8s and running it at scale are definitely appreciated.
|
|
#
¿
Jan 30, 2018 00:18
|
|
|
I guess the thing I'm not looking forward to is working with our platform engineering team to build out a k8s provisioner that actually suits our needs rather than using kops which gets us 80% of the way but leads down a dead end.
|
|
#
¿
Jan 30, 2018 14:32
|
|
|
Punkbob posted:Fwiw I think you can run stateless stuff in k8s. Shortcomings, aka not being able to whitelist security groups rather than cidrs. Kops isn't great, it's just an easy on-ramp. We run a pretty decent production load (anywhere between 1500-2000 instances at any time) and while all of our apps are stateless, moving to an opinionated cluster management system that doesn't lend itself well to CI use and doesn't slot in well with existing infrastructure seems...not good. Deploying services to k8s is one thing but building out and managing clusters at scale is another thing completely.
|
|
#
¿
Jan 30, 2018 18:38
|
|
|
Punkbob posted:I don’t disagree. K8s touchpoints with existing infra is bad and a lot of my issues that I deal with are related to that. I headcannon it as the k8s devs just expecting to be the only piece of infra to be one way and that’s what they build things towards. Yeah but that's fundamentally terrible unless you're selling a managed product. Existing infrastructure matters, especially if you're a company that's migrating from a different deployment method. We're likely not going to be able to do a full migration without writing our own k8s chef cookbook and deploying it in the same way we normally deploy clustered services. Sure it'll be nice to cut down on the number of different instance types and decrease the number of instances that run cold but it'll be a ton of work. Not sure how valuable it is at that point beyond speeding up deployments and marginally improving resiliency.
|
|
#
¿
Jan 31, 2018 04:34
|
|
|
fletcher posted:I was thinking maybe I need to create my users with known uid/gid before installing the RPMs, so that the RPM install doesn't create a user with whatever the hell ids it wants. As a general rule you want to make sure that whatever provisioning software you're running via Packer is idempotent. If the uids and gids are changing then you need to make sure that you always add service users ahead of time so that their identifiers are consistent.
|
|
#
¿
Apr 10, 2018 12:32
|
|
|
Methanar posted:Does anyone know of a proper guide for setting up triggers such that Jenkins will kick off jobs in response to This might help with interacting with jobs via GitHub: https://github.com/jenkinsci/ghprb-plugin
|
|
#
¿
Apr 11, 2018 23:39
|
|
|
Hadlock posted:Are you guys using 2FA on any of your internal services behind the firewall? We are rolling out LDAP to our internal services, but have the option to integrate U2F in a couple of them. Every one of our instances has Duo requiring 2fa for ssh access. Definitely do it.
|
|
#
¿
Oct 11, 2018 23:07
|
|
|
Docjowles posted:Current re:Invent status: Waiting in an hour long line to even register for the conference. Going to miss my first session. No food or coffee because everyplace that serves those also has an hour long line. Lol I waited for 3 minutes at the airport to get my badge.
|
|
#
¿
Nov 27, 2018 04:52
|
|
|
The best part about reinvent is ending up drinking on someone else's dollar in a private lounge. The conference itself is secondary.
|
|
#
¿
Nov 29, 2018 07:09
|
|
|
They fill two different purposes. Spinnaker is a deployment tool and Jenkins is a build tool. They can definitely be configured to work together and Jenkins can do some deployment stuff but you're better off looking at them separately.
|
|
#
¿
Dec 8, 2018 22:59
|
|
|
minato posted:The "immutable hosts" train of thought suggests that you configure it once on first boot with a tool like Ignition, and then you never touch it after that. If it's something like ContainerLinux then it'll auto-update itself with kernel upgrades. Any significant config change means nuking the cattle node and spinning up a new one. Which is totally fine if you've got a system like Kubernetes behind it to manage the rescheduling of workloads across nodes; not so much if you don't. And now you have 2 problems
|
|
#
¿
Dec 18, 2018 20:13
|
|
|
Methanar posted:Hot take: deploying kubernetes (properly) and throwing everything else into the trash is easier than making an existing system better. Hot take: deploying kubernetes (properly) and maintaining deployment systems on top of it takes more work (and reaps less rewards) than a mostly working existing system. This side of the industry loves new toys but gently caress me if kubernetes adoption for its own sake is the loving dumbest thing I've ever seen.
|
|
#
¿
Dec 19, 2018 01:41
|
|
|
Gyshall posted:Hi, I'm posting from the future from year 2321. Terraform 0.12 is still not released. Hi, I'm intensely demanding of an open source tool I don't pay for.
|
|
#
¿
Dec 21, 2018 00:34
|
|
|
StabbinHobo posted:cloudformation being terrible and terraform being worse (yep) is a lot of the reason i'm going in hard on k8s. Removes it from a lintable static-ish template language and puts it in a yaml hellscape from which you can never escape.
|
|
#
¿
Jan 3, 2019 23:57
|
|
|
LochNessMonster posted:I’m using git flow for our (small scale) infra repos and am wondering what kind of problems I’d he running into in the future and what alternatives there are. Care to elaborate on this? What's the difference between the head of develop and master? Infra should only have a finalised state and unless you're doing infra smoke tests using develop it's just another place where things can drift and conflicts can arise.
|
|
#
¿
Jan 21, 2019 18:14
|
|
|
Terraform has some template stuff and I've used it with cloud-init and launch configurations for basic config management type stuff but that's not really what it's designed to do and it'll be extremely frustrating if you ever need to scale out. You'll have better luck using Chef for all of the configuration stuff or even a 3rd party service like AWS SSM, Consul, or Zookeeper.
|
|
#
¿
Apr 21, 2019 23:02
|
|
|
|
| # ¿ May 2, 2024 04:19 |
|
|
Votlook posted:RabbitMQ or ActiveMQ, maybe Kafka if you are webscale. Reliable was one of the requirements
|
|
#
¿
May 13, 2019 23:04
|
|