|
Jenkins is the worst form of CI/CD – except for all the others that have been tried.
|
#
?
May 22, 2023 19:21
|
|
|
|
| # ? May 18, 2024 03:24 |
|
|
Honest question: what are the alternatives, and why are they better than Jenkins? At my current company, we use a Jenkins job DSL, so all the jobs are machine-generated, and refreshed hourly (so manual edits to jobs live up to an hour at most). Deploying a service involves running a handful of Jenkins jobs, but we use Spinnaker to orchestrate the deployments, so the various stages of a Spinnaker pipeline will invoke any particular Jenkins job. We started out with Jenkins only, and we added Spinnaker years later, mainly to allow anyone in engineering to do self-service prod deployments & day-to-day operations (scale up, terminate a bad node, etc.), and we wanted some these things to require approval from a second person on their team. In any case, this is just how the system organically evolved over time, but sometimes I do wonder if we were to do it all over again from scratch, what would that look like today?
|
|
#
?
May 22, 2023 19:29
|
|
|
terraform gzip'ing your lambdas
|
|
#
?
May 22, 2023 19:32
|
|
|
Alexei posted:In any case, this is just how the system organically evolved over time, but sometimes I do wonder if we were to do it all over again from scratch, what would that look like today? I've been liking using Gitlab to manage Pipelines, Environments, and Deployments
|
|
#
?
May 22, 2023 19:42
|
|
|
It's not really possible to give a no-nonsense answer to that question because these systems are never built or evaluated in isolation. The thing you always would have picked is the thing that you have the most experience with, the thing that lets you provide the most value for the least initial+ongoing effort. CI/CD systems are productivity boosters so they're always subject to this constraint. Theoretically though: If you need "approval from someone else on their team" there's a good chance that your DVCS already has a mechanism for this natively (gitlab pipelines, github actions, et al). If you need something "like Jenkins" there is AWX, which is basically just a feature superset of Jenkins with an actual API and an infrastructure designed on-purpose, this-decade. Of particular note here is the "workflow job" concept which features "approval step" as one of the available workflow node types. The one line summary of AWX is that its like Jenkins if all the plugins that make Jenkins usable weren't plugins but were just included in the product, along with some features that really should exist in Jenkins too like the ability to update the parameters for a job without pulling it from source and running it. If you have long running infrastructure jobs that would benefit from being purely code, there's temporal. I do think temporal is stupid and everything you can get from it you can get from AWX too, but, that requires a bit of line-between-reading and some ansible expertise.
|
|
#
?
May 22, 2023 20:25
|
|
|
I use azure devops
|
|
#
?
May 22, 2023 20:40
|
|
|
Wait hold on, awx is a web front end for ansible? I need to go play with this. I had been meaning to move my ansible homlab stuff to a dedicated VM/container for a bit now and that would be a pretty good excuse to get back around to it. Anything that doesn’t require 4 different plugins for basic functionality and 8 layers of abstraction to run commands is an improvement. I enjoyed my foray into GitHub actions a good bit and onto just learned you can run the executing agents on your own infra (another item for the pile). Speaking of, does anyone know of a githook trigger for an update/merge on a repo you don’t own? In a pinch you could just curl the version via cron and do diffs against what you have for the last updated version but something a bit more elegant would be nice. Warbird fucked around with this message at 20:47 on May 22, 2023 |
|
#
?
May 22, 2023 20:42
|
|
|
The Fool posted:I use azure devops  Dumb question but serious is there anything that Jekins does better or does something that ADO can't do?
|
|
#
?
May 22, 2023 20:55
|
|
|
It’s free OP
|
|
#
?
May 22, 2023 21:01
|
|
|
Jenkins would be fine if it had zero plugins. They cause all my headaches with managing it. Fuckin developers just can't stop coding. They code for a job and then are all "what if we coded for our code building too??!"
|
|
#
?
May 22, 2023 21:05
|
|
|
Warbird posted:It’s free OP ADO is free for 5 users and a single runner
|
|
#
?
May 22, 2023 21:05
|
|
|
Crosby B. Alfred posted:
Jenkins isnt an MS product
|
|
#
?
May 22, 2023 21:06
|
|
|
Warbird posted:Speaking of, does anyone know of a githook trigger for an update/merge on a repo you don’t own? In a pinch you could just curl the version via cron and do diffs against what you have for the last updated version but something a bit more elegant would be nice. Can't trigger on things outside of your project afaik, need to run on a schedule and do a diff, or a commit check or something along those lines
|
|
#
?
May 22, 2023 21:08
|
|
|
Github actions has gotten pretty good, there's not much it can't do, and all your developers already have access to the GHA logs (probably) so discoverability and visibility into how stuff works is really high, which seems to be a major pain points between developer/devops/ops The only plugin I typically have used with jenkins was improved global variables or whatever the hell they call it
|
|
#
?
May 22, 2023 21:12
|
|
|
jenkins is good job security but bad all other security
|
|
#
?
May 22, 2023 21:15
|
|
|
xzzy posted:Jenkins would be fine if it had zero plugins. They cause all my headaches with managing it. yeah poorly written / unmaintained plugins were the bane of my existence whenever I had to manage Jenkins. We still run it at my current job but it's not my problem to maintain it anymore 
|
|
#
?
May 22, 2023 21:24
|
|
|
vanity slug posted:jenkins is good job security but bad all other security Jenkins is all over job descriptions for govt IT work. I've been tempted to learn juuuust enough to put it on my resume.
|
|
#
?
May 22, 2023 21:26
|
|
|
Don’t do it OP, run. Gub’ment jobs are forever. Do you want to deal with Jenkins forever?
|
|
#
?
May 22, 2023 23:43
|
|
|
Warbird posted:Wait hold on, awx is a web front end for ansible? I need to go play with this. I had been meaning to move my ansible homlab stuff to a dedicated VM/container for a bit now and that would be a pretty good excuse to get back around to it.
|
|
#
?
May 23, 2023 00:08
|
|
|
Well what’s the go to central ansible control front end, if any? I don’t mind CLI but visualization is always nice.
|
|
#
?
May 23, 2023 00:26
|
|
|
This guy: https://www.redhat.com/en/technologies/management/ansible AWX is a CLI interface to the same.
|
|
#
?
May 23, 2023 00:51
|
|
|
it's not so bad. you have to stand it up first (theres a playbook that you edit an inventory file in to define some vars), and then its like - configure a Project - which has a Source (git repo with your playbooks in it) that it accesses through a Credential (probably a deploy token or ssh key) - you need an Inventory which has Inventory Sources that point to inventory files, plugins, scripts, etc., from your Project - you can now create Job Templates which reference your Inventory and which run playbooks pretty sure that's it. at work you would need to do a bunch of rbac poo poo too. there are ansible modules for setting all of this up but for your first time its probably better to learn by doing interactively.
|
|
#
?
May 23, 2023 02:19
|
|
|
The Fool posted:Jenkins isnt an MS product Yes, it isn't? I am out of the loop but my experience with Jenkins has been how in the hell is this so goddamn complicated?
|
|
#
?
May 23, 2023 04:29
|
|
|
12? 13? Year old enterprise software? With every nut and bolt and lever for every use case imaginable? What do you call a horse designed by committee? A camel
|
|
#
?
May 23, 2023 04:34
|
|
|
Vulture Culture posted:Have fun, trying to get AWX to do something useful is like working with Spacewalk back in the RHN Satellite days hard Disagree - "yeet JSON answers at the job templates API" make great glue for... everything? and Execution Environments are slick for running jobs on specific k8s clusters awx owns bones Junkiebev fucked around with this message at 03:10 on May 24, 2023 |
|
#
?
May 24, 2023 03:00
|
|
|
I'm trying to script against the Azure APIs for the first time, starting out with Python since that's my go to. Is this SDK actually the worst loving heap of poo poo ever developed by man, or am I doing something wrong? The documentation is completely opaque. It's just an enormous list of 4 billion methods across every service that exists, and if you don't already know what you're looking for gently caress you. Working with AWS and boto is like advanced alien technology from the year 7000 in comparison. I even tried to get ChatGPT to write me a basic code snippet and it could not produce code that actually ran. It kept getting method signatures wrong or trying to access attributes in responses that don't exist. Although it did finally help me identify which of the like 5 possible libraries for billing data was the right one to use. Maybe I'm just too used to AWS but this experience is embarrassingly bad for a multi-billion dollar tech giant. Are the libraries and documentation for other languages any better?
|
|
#
?
May 31, 2023 20:21
|
|
|
which api and what are you trying to do? yes it's all garbage but depending on what exactly you're doing there might be a better way
|
|
#
?
May 31, 2023 20:27
|
|
|
The Fool posted:which api and what are you trying to do? For our AWS accounts I've written a script that pulls monthly top-level cost data from every account in our organization for the past 6 months, as well as how much we are spending on each AWS service across the whole company, and dumps it to an Excel sheet. To track month over month changes in spend. This is basically to email around to executives who refuse to acknowledge the existence of any other data format. Granted I am much more familiar with AWS but this took me a couple hours from concept to completion to pull it all from the Cost Explorer API. My boss asked me to do the same for Azure and it's been a huge pain in the rear end. Which again could just by my Azure ignorance but the docs are not doing me any favors. Especially about billing data. I finally found the "azure.mgmt.costmanagement" Python package which seems to do what I want.
|
|
#
?
May 31, 2023 21:06
|
|
|
Azure APIs are, in places, poorly documented and confusing in my experience, and you’re experiencing the double whammy of also dealing with the cost management portion which may be totally unavailable to you depending on how you’re paying for Azure.
|
|
|
#
?
May 31, 2023 22:29
|
|
|
Azure's APIs are a 7-layer dogshit lasagne: - Unlike AWS where you can just get an access key attached to your IAM account and start using boto3 immediately, Azure forces you to register a service principal. But doesn't offer you a simple "look, I just want to give you a secret and then I make API calls", noooo, instead you have to define whether your app is mobile or does Oauth, and you need to supply a Redirect URI (!!), plus a zillion other confusing questions. - You can't just pass in the secret key and get an API session, you have to go through the whole song and dance of using the key to get a session token, and then using the session token to make the API calls. And handle storing a refresh token in a file somewhere so that you don't have to re-auth the next time you run the app. (In fairness, I think recent versions of the Python API tried to emulate the ease of boto3 by hiding most of this away) - Ok, so now you have a connection. What Python library do you want to use? Because there are multiple versions of each API, some with "preview" in the name despite being years old, and they're all simultaneously supported, so you need to pick a specific version. How you make sure the right version is installed and selected is basically magic. - Now you have a client instance and want to make an API call. So you visit the docs to understand the parameters, and they offer such gems as "subscription_policies: the subscription policies". Thanks! - More docs fun: inconsistent names! Tenant / Azure AD / Directory, they all mean the same thing and are used interchangeably. Sorry if ya didn't know! - Everything in Azure has a unique Object ID. However, sometimes this is hidden (in the Portal) and they show you a separate ID, e.g. both subscriptions and tenants have an ID that are different from their object ID. Some API calls use the object ID, some use the publicly visible ID. Trial and error till you figure out which one. - Unlike AWS, many API writes don't immediately take effect. You may need to install various waits() or polls() after you make a write to Azure, otherwise your program may hit a race condition. E.g. if you create a user and then try to grant that user a permission, the grant sometimes fails because the user hasn't finished being created in the background. So now your app is polluted with lots of polls and retries! - Don't even bother asking support unless you're on a premium support tier, because then you'll get redirected to Azure's awful 3rd party MindTree support, which is actually less then useless.
|
|
#
?
May 31, 2023 22:57
|
|
|
 When you are making a connection to Azure you are essentially creating an Enterprise Application or an object that represents an application. The thing is that it is basically no different than anything else because it's well an application. The whole Tenant / Azure AD / Office 365 thing is absolutely confusing but essentially think of it was a sub-domain under *.onmicrosoft.com. If you registered contoso.onmicrosoft.com you have full control over that and that is your directory. I agree on the rest of your points.
|
|
#
?
May 31, 2023 23:22
|
|
|
You can hit azure with postman and have it poo poo a cost csv on you iirc
|
|
#
?
May 31, 2023 23:49
|
|
|
Our flow for azure cost is basically
which is nasty but is basically the same as our aws cost reporting. We did get screwed when azure updated all the billing export formats not that long ago. The python documentation has been really bad, especially since you still have two different authentication libraries and you have to check which your api uses. I think most of it is auto-generated.
|
|
#
?
Jun 1, 2023 00:39
|
|
|
Docjowles posted:I'm trying to script against the Azure APIs for the first time, starting out with Python since that's my go to. Why are you going against the api and not using bicep or terraform?
|
|
#
?
Jun 1, 2023 01:04
|
|
|
One of those things is IAC and the other one is a joke Microsoft plays on their clients and im not sure how either would return billing data to be used elsewhere
|
|
|
#
?
Jun 1, 2023 01:16
|
|
|
New Yorp New Yorp posted:Why are you going against the api and not using bicep or terraform? I don’t know what bicep is. Is there a terraform resource for “run an arbitrary billing query on demand”? I don’t think what I’m doing is insane at all in AWS. Cost Explorer has a very easy api for exactly this. But this is my first real exposure to Azure Docjowles fucked around with this message at 01:45 on Jun 1, 2023 |
|
#
?
Jun 1, 2023 01:42
|
|
|
there is the azapi provider which lets you hit arbitrary end points, but I don't think it works on the billing api and is super clunky to use anyway the python library is your best bet for your use case
|
|
#
?
Jun 1, 2023 01:44
|
|
|
Docjowles posted:I don’t know what bicep is. Is there a terraform resource for “run an arbitrary billing query on demand”? Just checking that you're using the right tool for the job. API is probably correct.
|
|
#
?
Jun 1, 2023 02:13
|
|
|
If you were tasked with overhauling monitoring and alerting + improve visibility into a highly real-time system, what kind of KPIs would you select for phase 1 of the project, and what would you push out to phase 2 or beyond Also thoughts on how to minimize alert fatigue
|
|
#
?
Jun 1, 2023 05:53
|
|
|
|
| # ? May 18, 2024 03:24 |
|
|
Hadlock posted:If you were tasked with overhauling monitoring and alerting + improve visibility into a highly real-time system, what kind of KPIs would you select for phase 1 of the project, and what would you push out to phase 2 or beyond define SLOs, implement APM if you don’t already have it, alert when you are projected to cross those SLOs/SLAs and try to avoid alerting on literally anything else. Capture metrics everywhere you can and use them liberally for auto scaling/autohealing but who gives a poo poo about resource usage unless it’s breaking a budgetary threshold? HA obviously. Make sure your services are idempotent or implement checkpoints to handle failures mid request/transaction, and ensure you have retries where feasible for invoking your dependencies. SLOs for your service, its subcomponents, and its dependencies need to come first. Everything else flows from that. Your golden signals are request rate, latency, and error rate. The latter in particular can be defined in many different ways and will probably differ amongst components of your service, which is fine. APM and telemetry (logs/metrics/traces) is the ideal way you differentiate between those but failing that you’re probably storing state somewhere and can leverage it if APM is too great an initial lift. Ensure developers and service owners are empowered to act when problems occur, and that you have good logging and observability tooling to quickly identify problems with your system’s internal state. Read Charity Majors’ blog. The Iron Rose fucked around with this message at 06:34 on Jun 1, 2023 |
|
#
?
Jun 1, 2023 06:26
|
|
Cat Army