|
I sent a firewall exemption ticket in to get our puppet compile masters the ability to talk to the agents today. It’ll only be 2 weeks. This poo poo was supposed to go live last October. 
|
#
?
Jun 23, 2018 05:41
|
|
|
|
| # ? May 15, 2024 04:08 |
|
|
ratbert90 posted:I set up my first Jenkins server today with AWS AMI slaves and it’s pretty slick. Took me 8 hours because I only had passing knowledge of AWS and 0 experience with Jenkins. So far I am very happy with it. Nice. I only started working with Jenkins about 2 weeks ago but I'm digging it.
|
|
#
?
Jun 23, 2018 09:05
|
|
|
Report back in 4 months for how you feel about Jenkins
|
|
#
?
Jun 23, 2018 12:08
|
|
|
Methanar posted:Report back in 4 months for how you feel about Jenkins Buddy, the horrible poo poo I have have seen over the years as an embedded Linux engineer, nothing Jenkins does could even phase me.
|
|
#
?
Jun 23, 2018 12:23
|
|
|
Now you gotta put a bunch of sketchy plugins on there, and try to guess which ones will never be updated ever again.
|
|
#
?
Jun 23, 2018 19:42
|
|
|
Helianthus Annuus posted:Now you gotta put a bunch of sketchy plugins on there, and try to guess which ones will never be updated ever again. That's easy. All of them.
|
|
#
?
Jun 24, 2018 03:23
|
|
|
Embrace the giant red box of security shame whenever you click Manage Jenkins Warnings have been published for the following currently installed components (list of all currently installed components follows)
|
|
#
?
Jun 24, 2018 04:10
|
|
|
Docjowles posted:Embrace the giant red box of security shame whenever you click Manage Jenkins Do you want to adopt this plugin?
|
|
#
?
Jun 24, 2018 04:18
|
|
|
Never install a Jenkins plugin unless it is a hard requirement for what you are trying to do. And even then give a serious consideration to an absurd rube goldberg machine involving a webserver written in bash first.
|
|
#
?
Jun 24, 2018 05:22
|
|
|
Methanar posted:Report back in 4 months for how you feel about Jenkins Care to be more specific so that I at least know what I'm getting into?
|
|
#
?
Jun 24, 2018 13:55
|
|
|
The more plugins and builds and agents you push through Jenkins the shittier it gets. Itll randomly start losing connections to agents, freeze up randomly and just in general be a pain in the rear end. And once you start having issues with it crashing devs will basically scream at the top of their lungs when it does break and make it a tier 1 production app basically, that doesn’t actually get the priority from the business to fix. This becomes an especially vicious circle as everything you look to migrate to has to then support the rats nest of poo poo built in groovy which a build system shouldn’t actually be doing. I’d also add that most other build systems now use an entirely different model because Jenkins is bad at actually building poo poo but ok at task running? So swapping to something that actually works is a pain in the rear end because you need to redo everything in that new model. Edit: also its security is terrible and is often a great place to start recon or just plain infiltrating a network.
|
|
#
?
Jun 24, 2018 15:47
|
|
|
I’m setting up a side project at home with about 10 different microservices. Code is managed in Gitlab and consists of python apps, haproxy, webservers and databases all running in docker containers. What would you guys pick for a CI tool to deploy this? I could go with Jenkins as that is what I’m using at work but I’d rather learn something new that is not a pos.
|
|
#
?
Jun 24, 2018 16:12
|
|
|
LochNessMonster posted:I’m setting up a side project at home with about 10 different microservices. Code is managed in Gitlab and consists of python apps, haproxy, webservers and databases all running in docker containers. Gitlab CI isn’t bad fwiw. If you are looking for dirt cheap and unmanaged google container builder is decent. But doesn’t have native gitlab support. If windows teamcity. Otherwise self hosted best options are drone(simpler) or concourse(more full featured)
|
|
#
?
Jun 24, 2018 17:06
|
|
|
Gitlab CI is the stones. Give it a try.
|
|
#
?
Jun 24, 2018 23:21
|
|
|
Currently migrating from Bamboo to Jenkins because our company is completely insane, we have some goofy license that allows unlimited bamboo server installs, but then we have to pay $$$ per agent per year. So we have 16+ bamboo servers, each with 32 cores and 192GB ram and 0 agents When we ran in to a problem (turned out we hit the linux user process limit for the bamboo user) Atlassian sent us an email something like "we have never heard of anyone ever running bamboo in this configuration, we suggest you use more agents than just the primary bamboo server" Currently only using jenkins to run bash to build containers and deploy them. If anything needs some goofy jenkins plugin, time to add a trigger to jenkins and move that process outside of jenkins.
|
|
#
?
Jun 24, 2018 23:21
|
|
|
That sounds crazy. They did add docker support to the recent Bamboo but I don’t know how that works with agents and pricing. Would be convenient though to just run a docker command and exit instead of having an agent just sit there doing nothing most of the time. We have Jenkins running a few things at my current gig but when I found out they get Atlassian products for free I set that up because we used it in my last job to good results. Plus I’m a sucker for the integration with Jira and Bitbucket and shared user management.
|
|
#
?
Jun 24, 2018 23:42
|
|
|
VSTS is cool and good if on-prem is not a requirement.
|
|
#
?
Jun 25, 2018 00:01
|
|
|
The Fool posted:VSTS is cool and good if on-prem is not a requirement. And can be installed on-prem if it is, although of course it only runs on Windows/MS SQL. The build/release agent is cross-platform, though.
|
|
#
?
Jun 25, 2018 00:08
|
|
|
freeasinbeer posted:Gitlab CI isn’t bad fwiw. If you are looking for dirt cheap and unmanaged google container builder is decent. But doesn’t have native gitlab support. Gyshall posted:Gitlab CI is the stones. Give it a try. Gitlab CI it is, project is already in Gitlab so makes sense. Heard some good things about it before so let’s see how this works.
|
|
#
?
Jun 25, 2018 13:27
|
|
|
It’s pretty good. Gitlab is doing good poo poo. I don’t think I’d host with them but I’d take gitlab enterprise in house over GitHub
|
|
#
?
Jun 26, 2018 04:49
|
|
|
Yeah in house takes like two minutes and is p. simple to get up and running on k8s as well.
|
|
#
?
Jun 26, 2018 16:15
|
|
|
Gitlab is good, I have yet to be burned by a gitlab update. Their omnibus package installs everything to a non-system path and uses an embedded Chef to keep everything sorted nicely. On the other hand, updating Jenkins is a risky thing to do, I guess mostly because of the decrepit plugins I have installed.
|
|
#
?
Jun 26, 2018 20:55
|
|
|
things I have learned from jenkins issues in the last five months: 1. don't ever update, or install, jenkins
|
|
#
?
Jun 26, 2018 23:28
|
|
|
Ask me about upgrading the kubernetes plugin for jenkins. Also ask me about trying to build containers with a Jenkins installation that is itself running within containers on kubernetes. Don't
|
|
#
?
Jun 27, 2018 00:45
|
|
|
It's containers all the way down
|
|
#
?
Jun 27, 2018 00:46
|
|
|
The Fool posted:It's containers all the way down Except containers in containers isn't actually containers in containers because docker-in-docker doesn't exist because even the sub containers are still sharing the same host kernel as everything else docker in docker is really docker beside docker because everything is actually sharing the same docker daemon but it becomes bad because containers started by anything other than kubernetes are outside of the control of the resource scheduler and worse still is trying to pass around the docker unix socket because you're like 5 filesystem abstractions deep (file system abstractions are also side-by-side rather than actually nested but lol have fun parsing docker inspect to find where the gently caress anything actually is on the real host root filesystem) also good luck trying to get containers that are not managed by kubernetes, but still on a kubernetes host, to have any sort of network connection because you won't have the nice resource scheduler hooking you up to whatever flannel/calico/weave CNI you've got gently caress
|
|
#
?
Jun 27, 2018 01:14
|
|
|
The Fool posted:It's containers all the way down We just rolled out a private dns system that's dns servers as containers, slave dns server containers at each office, aws doesn't do udp load balancers so those are containers, and then dns-exporters in containers to validate everything's working as expected. Monitored by Prometheus/Grafana which are also containers.
|
|
#
?
Jun 27, 2018 06:13
|
|
|
Have any of you done a sort of "terms and conditions" for things like cloud adoption? Our internal teams are pushing for more autonomy and as such are wanting to get their own AWS accounts and manage their infrastructure, which in theory sounds great. But when they need to VPC peer into another teams account or connect down to our datacentre something has to be in place to ensure they're not going hog wild with spend, not create blatant security risks etc. I'm trying to avoid a heavily prescriptive top-down approach to policy as that slows everybody down, but management want to be seen to have a handle on things, or at least make sense of it all. I've started work on a set of tools that descend from our root account and ensure simple things are covered; do teams have a budget, are resources tagged, etc. but not sure where to go from here in terms of making this all fit together cohesively.
|
|
#
?
Jun 28, 2018 13:55
|
|
|
Cancelbot posted:Have any of you done a sort of "terms and conditions" for things like cloud adoption? Our internal teams are pushing for more autonomy and as such are wanting to get their own AWS accounts and manage their infrastructure, which in theory sounds great. But when they need to VPC peer into another teams account or connect down to our datacentre something has to be in place to ensure they're not going hog wild with spend, not create blatant security risks etc. We have a few sets of things like that. Security has an IAM account with basically god level read only access to the accounts and run real time monitoring of changes. There's a cloud committee that discusses how to play well together but it's at a mass scale, multiple loosely related corporations under one ownership touching every cloud provider out there.
|
|
#
?
Jun 28, 2018 14:15
|
|
|
Cancelbot posted:Have any of you done a sort of "terms and conditions" for things like cloud adoption? Our internal teams are pushing for more autonomy and as such are wanting to get their own AWS accounts and manage their infrastructure, which in theory sounds great. But when they need to VPC peer into another teams account or connect down to our datacentre something has to be in place to ensure they're not going hog wild with spend, not create blatant security risks etc. I work in Azureland but I'm sure analogous things exist in AWS. 1) Policies are in place to restrict some things (can't create resource types X,Y,Z, VMs can't be in the "super expensive" class, etc). There's a well-defined, relatively red-tape-free process to say "Hey, I need an exception". 2) All changes are made via ARM templates that are stored in source control and are applied via a continuous delivery pipeline. All changes go through PRs and someone from the corporate cloud group is involved in the PR. Dev/test subscriptions exist to test ARM templates and/or tweak things through the portal with impunity. 3) Some things (generally, networking-related stuff like VPN connections and virtual networks for VMs) are controlled by corporate. If your VM needs to be able to talk to X, here's the vnet you connect it to. If you want to set up an elaborate weirdo internal network that doesn't interact with anything else, go nuts, do whatever you want.
|
|
#
?
Jun 28, 2018 14:27
|
|
|
Methanar posted:<roomba stuck in a corner>
|
|
#
?
Jun 28, 2018 14:35
|
|
|
It's a tough thing to balance. We had a dev team successfully politic their way to their own AWS account with full control over it, minimal oversight from the operations team. At some point they requested an increase in the EC2 instance limit to 1,000 for like 10 different types in multiple regions. Then someone with admin access committed his access key to GitHub, and we spent $17,000 on buttcoin miners in about 15 minutes (they hadn't enabled 2FA, naturally). We caught it immediately and shut it all down, and AWS thankfully refunded most of that. It was great. They do not have YOLO-tier access anymore. I'm very receptive to the argument that teams need access to spin up and manage stuff. If you aren't going to let them move fast, why are you paying a huge premium for the cloud? But definitely put some guardrails on. There's a medium somewhere between old-school Ops as gatekeepers who say no to everything, and giving the root login to people with zero security or operational background. Interested to hear how others have threaded the needle, too.
|
|
#
?
Jun 28, 2018 15:03
|
|
|
edit: double post 
|
|
#
?
Jun 28, 2018 15:03
|
|
|
Thanks! Yeah we're going for "if its inside your VPC, go hog wild!" it's more if they want to directly interact with networks and customer data do we want to make sure they're being careful.
|
|
#
?
Jun 28, 2018 15:25
|
|
|
Let devs go hog wild in the AWS accounts for the staging environments and have production deploys gated by the ops team's automation? 
|
|
#
?
Jun 28, 2018 16:21
|
|
|
Anyone here fooled with Nirmata before?
|
|
#
?
Jun 28, 2018 16:31
|
|
|
Cancelbot posted:Have any of you done a sort of "terms and conditions" for things like cloud adoption? Our internal teams are pushing for more autonomy and as such are wanting to get their own AWS accounts and manage their infrastructure, which in theory sounds great. But when they need to VPC peer into another teams account or connect down to our datacentre something has to be in place to ensure they're not going hog wild with spend, not create blatant security risks etc. I am loathe to use it unnecessarily, but this is where the "D" word comes in. The companies I've consulted for are inserting "devops" there to have some automation in place to reflect the policies laid out from management. There's nothing magical about it, but in AWS, a simple answer is to have something like Service Catalog products ready to go with choices for things like instance class and VPC. This works pretty well for SMBs, or as a starting point for any sort of self-service initiative. Scaling that up, I helped implement an API for a customer who wanted to be able to create new VPCs/AWS accounts with a single HTTP call, we used Step Functions to handle the workflow. Of course they had rather heavy IAM in place to safeguard these new treasures. I can expound a bit more on this stuff, but I'm limited somewhat by NDA. The bottom line is that you need to have those policies reflected by automation/code or else they aren't worth a hill of beans. Anything not enforced in this way leads to keys on Github and $68k miner happy hour. Not only that, but self-service avenues mean that you can make it easier for teams to adopt those methods, which reduces the burden on your ops team to provision resources and gives them more time to roll out cool stuff like this.
|
|
#
?
Jun 28, 2018 20:39
|
|
|
Look at integrating your corporate auth source with your AWS resources, as well as policy tags. We destroy resources every Friday that don't have the appropriate tags on them, as well as limit AWS provisioning to certain AD groups. Also a good project to extend the proverbial devops Olive Branch to your traditional IT folks.
|
|
#
?
Jun 28, 2018 21:27
|
|
|
Vanadium posted:Let devs go hog wild in the AWS accounts for the staging environments and have production deploys gated by the ops team's automation?
|
|
#
?
Jun 29, 2018 03:34
|
|
|
|
| # ? May 15, 2024 04:08 |
|
|
Grafana natively supports AWS cloudwatch as a datasource out of the box, all it needs is a read-only billing IAM key to get started. We have it on our main monitoring display and it's nice to have a visual representation of how much you spent last month vs your rate of spend this month. We're moving pretty rapidly in to AWS from bare metal world and it's easy to leave extra poo poo on, or over-provision. We jumped from ~$7K spend to $14K spend and were able to dial it back by watching the graph. Boss man also likes it for budgeting as it demonstrates a pretty linear growth rate in the sawtooth of each month at the 6 month zoom level that the finance/budget/CFO guys like and gives everyone a pretty warm fuzzy that spending is under control click to embiggen  https://grafana.com/dashboards/139
|
|
#
?
Jun 29, 2018 06:06
|
|