|
IMO Basically your entire company should, if at all possible, use the same ticketing system. I recognize that means that some uses are going to be suboptimal but that's not nearly as much of a pain in the rear end as the alternative.
|
#
?
Oct 5, 2023 23:35
|
|
|
|
| # ? May 21, 2024 15:01 |
|
|
SurgicalOntologist posted:It's probably relevant context that our "product org" is... 2 people? And we have 12 engineers, that's if you count 3 students we support in their PhDs that are doing longer-term R&D. Yeah that's important context Many companies ago, I was employee #26 and the management & product teams were using.... asana and... m something, while engineering was using JIRA to some minimal extent. Thankfully about 6 weeks after I started they finally ripped the bandaid off and moved everyone over to JIRA and cancelled their asana + m__ subscriptions. We scaled to 150 people over the next two years it was pretty successful, except when someone from product would try and reformat all 27 jira projects to use the same XYZ custom fields and delete half the tickets (this last part happened at three different companies take heed Sounds like you're 6-12 months away from unifying your ticketing system, good luck
|
|
#
?
Oct 5, 2023 23:53
|
|
|
Falcon2001 posted:IMO Basically your entire company should, if at all possible, use the same ticketing system. I recognize that means that some uses are going to be suboptimal but that's not nearly as much of a pain in the rear end as the alternative.
|
|
#
?
Oct 6, 2023 05:16
|
|
|
Hadlock posted:try and reformat all 27 jira projects to use the same XYZ custom fields and delete half the tickets (this last part happened at three different companies take heed Yeah our Jira instance is customized to hell and has a billion plugins and like 15 years of history. It's barely recognizable and a bear to manage and do upgrades on. Every few years some manager decides they will be the conquering hero who tames our Jira. Then they start looking into the nuts and bolts of making it happen and realize removing this one custom field breaks 500 other things. Or this super expensive plugin that everyone else hates is critical to Finance closing the books each month or something. They inevitably surrender in despair. 0/10 do not recommend giving every team in a large company autonomy to do whatever the hell they want with their Jira projects. I feel great pity for the IT group who is eventually going to have to deal with porting this poo poo heap to Jira Cloud.
|
|
#
?
Oct 6, 2023 17:42
|
|
|
When I worked at Microsoft there was a period of time where my team has six different ticketing systems to use that had zero interoperability and it was a loving nightmare. I also used Visual Studio Online / Azure DevOps as our incident management tooling, so that was an interesting experience. Before I left Azure had done a reasonably good job of browbeating everyone into the same internal tool for ticketing, which was a huge relief, although it didn't include task management, so you still basically had ADO + InternalTicketingSystem. My new BigTechCompany has literally one with different frontends for task/incident and while it's also not perfect it's such a better system. (Side note: obviously customer-facing ticketing is probably not something you can easily integrate with your internal one for safety reasons so ignore that.)
|
|
#
?
Oct 6, 2023 18:33
|
|
|
We have a customer facing system, and an "agile" ticket tool. And one of our groups insists every interaction (even with internal tools) flows through a request form in their customer facing system, which creates a ticket in that system. Then they manually copy the details into our agile tool, put a reference into the customer ticket, and resolve the ticket.
|
|
#
?
Oct 6, 2023 19:46
|
|
|
FISHMANPET posted:We have a customer facing system, and an "agile" ticket tool. And one of our groups insists every interaction (even with internal tools) flows through a request form in their customer facing system, which creates a ticket in that system. Then they manually copy the details into our agile tool, put a reference into the customer ticket, and resolve the ticket. lol 
|
|
#
?
Oct 6, 2023 22:08
|
|
|
any backstage users here?
|
|
#
?
Oct 11, 2023 09:42
|
|
|
my homie dhall posted:any backstage users here? Not sure if this was a good thing or a bad thing though
|
|
#
?
Oct 11, 2023 09:46
|
|
|
it's the jenkins of this decade
|
|
#
?
Oct 11, 2023 10:39
|
|
|
vanity slug posted:it's the jenkins of this decade
|
|
#
?
Oct 11, 2023 17:40
|
|
|
someone tell me why nomad is bad
|
|
#
?
Oct 11, 2023 18:42
|
|
|
We evaluated Backstage but ended up buying https://www.cortex.io/ for a service catalog / developer portal / current buzzword. The pricing wasn't too bad once we did the usual enterprise.txt dance of beating them down to 20% of the original quote. We are terrible about tracking and publicizing service ownership and dependencies so it's been very useful. As always you can't fix a culture problem with a tool, though. If there is zero consequence for not onboarding your services and not complying with standards, then guess how many people are gonna do it. Backstage seems fine for what it is but it's also kind of a hassle to operate and maintain. We felt like it was a better use of resources to pay a company whose entire business is building a service catalog, rather than dedicate full time headcount to "Keep Backstage Running" engineers. Which is refreshing for us. I feel like 10 years ago this company would have found 1 single aspect of Backstage they didn't like, started a private fork we have to maintain forever, and then the person responsible for this choice would leave the company 6 months later. We're learning 
Docjowles fucked around with this message at 18:59 on Oct 11, 2023 |
|
#
?
Oct 11, 2023 18:42
|
|
|
The Fool posted:someone tell me why nomad is bad It’s a lot harder to find and hire experts in nomad compared to kubernetes.
|
|
#
?
Oct 11, 2023 18:50
|
|
Cat Army
|
The Iron Rose posted:It’s a lot harder to find and hire experts in nomad compared to kubernetes. https://twitter.com/QuinnyPig/status/1712146775644213290
|
|
#
?
Oct 11, 2023 18:55
|
|
|
I like Corey but he really seems to despise k8s to an irrational extent
|
|
#
?
Oct 11, 2023 19:02
|
|
|
im dangerously close to falling into some koolaid here
|
|
#
?
Oct 11, 2023 19:11
|
|
|
Docjowles posted:I like Corey but he really seems to despise k8s to an irrational extent I just walked past him in the hallway
|
|
#
?
Oct 11, 2023 19:49
|
|
|
The Fool posted:someone tell me why nomad is bad the lifecycle thing is half-baked - it only supports pre/main/post, so if your dependency chain is more than 3 layers deep, you need to hack something up yourself templates and secrets (= dynamically generated config files) can only be written into a special $NOMAD_SOMETHING_DIR inside the container (or to an envvar), so hope all your services can be told to load config from an arbitrary path/envvar or you'll have to add a startup one-liner script that moves it the podman driver doesn't support dns passthrough. also it really should be built-in at this point the cli is missing a "nomad job $job latest-alloc status" command, I had to make a script that greps for the most recent alloc id. such pain those are literally my only criticisms of it so far. it's been wonderful otherwise
|
|
#
?
Oct 11, 2023 20:59
|
|
|
Docjowles posted:I like Corey but he really seems to despise k8s to an irrational extent It's a sentiment I've seen a lot which is basically that k8s seems to be deployed more often than its appropriate, and usually because the engineer(s) doing so are doing it because its the hot thing to do for their careers rather than what's best for the organization. Honestly I have no idea how true the sentiment is, but I did have a coworker running a k8s cluster for a single application running in a single container that never had enough demand to warrant even a second instance of the app, so I have seen it.
|
|
#
?
Oct 11, 2023 21:59
|
|
|
Kubernetes is great, but it absolutely demands opinionated subject matter experts (I.e. a dedicated devops team), which isn’t always feasible. There are many good alternatives for services you’re authoring yourself, from your standard serverless functions to offerings like ECS/cloud run/app service. There’s hidden gotchas that you won’t see coming, like the fact that swap doesn’t exist so you need to make sure your memory intensive stateful services can recover from getting OOMKilled in the middle of a transaction. That being said, the ecosystem is extraordinarily broad, tons of poo poo integrates with it, vendors often author their own helm charts which makes deployments easy, and it does bin packing very very effectively. It’s a great product and so much better than running services on VMs it’s not even funny. Also Corey is great and bought drinks for everyone at the Monitorama after party. I made a point of not asking him about anything technical, but the man is clearly brilliant as he immediately ducked out when the random drunk 20 year old girls started hanging out at our table.
|
|
#
?
Oct 11, 2023 22:57
|
|
|
The nice thing about kubernetes is that when your kubernetes guy leaves, you can hire Kubernetes contractor #12345 to come in and maintain it. He will read the helm chart, look at the terraform and know exactly where and how the app is deployed. If you ever deploy a second app, just copy-paste the helm chart and change the container name and maybe namespace(s) and now you have two apps deployed the same way on the same cluster. If your developers push out code to an elastic beanstalk thing, fine, I guess. And then another developer starts If one person can own being opinionated about how you deploy most of your services/code that's great but more often than not nobody owns that and it degrades to lord of the flies operational chaos over a couple of years as more I talked to one company recently they're a team of 4 devs on EBS with like 3 services, but no IaC and ok with moving to kubernetes because they plan on ramping up to ~10 services over the next two years and managed kubernetes is well inside their budget as it gets them to operational maturity much faster Doesn't help that permissions with aws->eks is kind of sloppy; gke is a dream to work with by comparison
|
|
#
?
Oct 11, 2023 23:19
|
|
|
yeah the value of having the shared lingua franca in k8s should not be underestimated, which is why we don’t deploy much to cloud run/app service/ecs as the expense of running k8s is already baked in. I’d still use serverless functions over k8s whenever remotely feasible though.
|
|
#
?
Oct 11, 2023 23:35
|
|
|
I've been looking at a few of the SecretOps tools and one thing I am not understanding is, what does the process look like for defining a new secret and rolling it out to all the necessary environments? Is the operations team just supposed to look at the dashboard before a release and say "oh poo poo there's a red X in the UI here where the secret value hasn't been specified yet" and then they specify the value and then the release proceeds?
|
|
#
?
Oct 11, 2023 23:45
|
|
|
SecretOps feels like a quintessential zero interest rate phenomenon. Just build it into your normal test/release cycle. You’ll have a secret store - whether you roll your own or use a managed service. Your deployment pipeline will probably fetch from that secret store and either bake it into your container as an envvar or your application can query from a secret store and fetch it during runtime. Both have obvious tradeoffs. In either event, adding new secrets is something that is defined in code and deployed as code, and your standard merge request reviews and integration tests should catch things like “oops forgot to set foo:bar in prod” easily enough. As far as updating secrets goes that’s so service specific it’s not really worth building a product out of, but hey maybe I’m wrong. Kubernetes will at least automatically restart your pod if the mounted secret object changes!
|
|
#
?
Oct 12, 2023 00:01
|
|
|
The Iron Rose posted:SecretOps feels like a quintessential zero interest rate phenomenon. Thanks for the reply. I think this is the part that I'm having a having a trouble visualizing what it looks like: quote:In any event, adding new secrets is something that is defined in code and your standard merge request reviews and integration tests should catch things like “oops forgot to set foo:bar in prod” easily enough. How does that work, if the value of the secret is not something that is being committed to a repo?
|
|
#
?
Oct 12, 2023 00:04
|
|
|
In my experience the developer is responsible for that, and ops ends up doing a lot of hand holding New secrets would be loaded into the test environment using a sops encrypted yaml file, test-secrets.yaml or whatever, then when they're ready they'd modify qa-secrets.yaml, and finally roll out the change to prod.yaml Helm will fail if you try to render a template with a missing value, so you will catch it then. I'm not sure how you ensure a process is validated, I guess you could do a line count of the config files or something and block PR merge if they don't match. This is largely a process based problem imo, I'm curious about automating this too, but I'm not sure how you automate "don't forget to add the secret to prod" without some kind of secret intake process
|
|
#
?
Oct 12, 2023 00:09
|
|
|
fletcher posted:Thanks for the reply. I think this is the part that I'm having a having a trouble visualizing what it looks like: So I’ll describe two approaches. One we commonly use for kubernetes apps, and one we commonly use for everything else. They are actually the same approach when you get down to it. Fundamentally, you have an identity associated with your running service, and have code inside your service that queries your secret store and authenticates with its identity to retrieve and decrypt the secret. At that point, it’s up to the service to decide what to do with it. We use gitlab runners, but literally anything that can run arbitrary code as part of a CI pipeline will do fine here. Each gitlab runner is instantiated as a kubernetes pod, so they are ephemeral, and have an identity (either the k8s service account or a cloud specific workload identity). The runner will authenticate to and query your secret store for the secret, and then do whatever it needs to do with it (for example, authenticating to our private container registry before pushing an image). For kubernetes deployments, we use helm for our deployments, and we use helmfile (https://github.com/helmfile/helmfile) to orchestrate coupled releases. We often have a need to provide secrets as environment variables or mounted volumes for vendors, and it can be convenient for our own authored services as well. We do this using the helm-secrets plugin (https://github.com/jkroepke/helm-secrets), whereby we store a reference to the secret in code. The CI pipeline will take that reference, decrypt the secret, and pass the decrypted value to the data field for a kubernetes secret object. This secret object would be one Helm release, or the Helm chart needs to define a secret object to store its secrets. The chart for the application that needs the secret can then reference the secret object. Note that Helm treats kubernetes secret data as sensitive, so the plaintext secret is not exposed in job logs (conversely, sensitive Helm values that aren’t referencing secret objects *are* shown in plaintext). The application pod can then reference the secret as an environment variable or mount it as a volume as need be, which would be defined by the Helm chart and how it passes values to the k8s template. The above is specific to kubernetes and helm but the concept can be applied anywhere. you store a reference to the secret in code, and then whatever is executing that code (your pipeline jobs, the service during runtime, etc) will retrieve the secret from your secret store and put it where it needs to go (into kubernetes’ secret store, or as input into a request, or whatever you need to logically do with the decrypted secret). Only the reference is exposed in your VCS. That reference can be SOPS encrypted, or it can literally just be the URL (or url fragment) where you’d retrieve the secret over the network, or if it’s a Kubernetes application it can just be the secret and key name. As a side note, while you *could* in theory store secrets as an envvar or file in your container image, i would highly recommend otherwise. Anyone who can pull the container can access the secret, and often that’s a broader group than anyone who can access your application during runtime or the backing secret store. The Iron Rose fucked around with this message at 00:31 on Oct 12, 2023 |
|
#
?
Oct 12, 2023 00:26
|
|
|
I'll preface this by saying that I wasn't responsible for janitoring the Nomad/Consul/Vault servers themselves, but helped lead the move of a system with hundreds of developers and tens of millions of users to that stack a few years ago at a former employer. We weren't the biggest Nomad deployment but definitely one of the larger ones. Outside what felt like a run of self-inflicted teething problems, it was mostly trouble-free. I have a hard time blaming Nomad for not putting firewall rules in place to keep production from kissing test when misconfigured one day, for example. I did bump into the lifecycle issue mentioned above and we ended up changing some things to fit better with Nomad's model where possible and hacking around it with scripts where we couldn't. It had some rough edges a few years ago, sounds like that's still the case. I would still recommend a serious look at Nomad, for all the rough edges in development it was solid from an operational standpoint. Task Drivers were a big deal for this company, it meant they could orchestrate legacy workloads on existing hardware and then migrate to upgraded hardware transparently without downtime, which was a huge win given the SLAs and penalties involved for unscheduled downtime. The ability to orchestrate non-container workloads and write plugins for runners that didn't exist natively was a gamechanger and was ultimately why Kubernetes wasn't adopted there.
|
|
#
?
Oct 12, 2023 01:20
|
|
|
The Iron Rose posted:So I’ll describe two approaches. One we commonly use for kubernetes apps, and one we commonly use for everything else. They are actually the same approach when you get down to it. Thanks for the detailed write-up, this is super helpful. This seems to thoroughly cover reading the secrets, but what's the process around writing the secret values to the secret store?
|
|
#
?
Oct 12, 2023 01:44
|
|
|
fletcher posted:Thanks for the detailed write-up, this is super helpful. This seems to thoroughly cover reading the secrets, but what's the process around writing the secret values to the secret store? There's no real way around either having a human do that one, or having some service generate and upload the secret in question.Very much depends on what you're building, and most of the time that's a human doing so. Implicit in the concept of a secret store is that the values change semi-infrequently. Hashicorp Vault, to its credit, has a lot of support for generating truly ephemeral credentials, which it largely does by operating as an intermediary that generates ephemeral tokens on your behalf. Similarly, there are various cloud provider offering that do similar things - Azure AD App Proxy, GCP IAP/auth proxy, etc - and you can always implement OIDC if you want a truly modern authentication protocol for secure communication between services. Obviously, it's not possible to do that all the time. Sometimes you just need to store an API key or client secret somewhere and there's often no better way to do so other than to have a human with write (not necessarily read) access do it. The Iron Rose fucked around with this message at 02:48 on Oct 12, 2023 |
|
#
?
Oct 12, 2023 02:44
|
|
|
Does Nomad have the ability to inject it's child processes with secrets from Vault yet? I haven't looked at it in a really long while, but modern k8s stuff with Vault is so drat nice where you can do things like generate k8s secrets from an operator or inject secrets into pods through various forms. Without really knowing that side of the ecosystem, it really feels like there is way more built around k8s.
|
|
#
?
Oct 12, 2023 03:28
|
|
|
they've been really playing up the integration with vault and consul this week so maybe it can now?
|
|
#
?
Oct 12, 2023 03:42
|
|
|
The Fool posted:they've been really playing up the integration with vault and consul this week so maybe it can now? Imagine they'll loving gut you on per-user licenses.
|
|
#
?
Oct 12, 2023 03:48
|
|
|
drunk mutt posted:Imagine they'll loving gut you on per-user licenses. even worse, it's a per identity license for vault. vault's a fantastic product but a quarter million+ a year for support and the enterprise features is pretty steep. it's classic hashicorp: make the free product amazing, and the enterprise version juuuuuuust too expensive.
|
|
#
?
Oct 12, 2023 04:13
|
|
|
drunk mutt posted:Does Nomad have the ability to inject it's child processes with secrets from Vault yet? I don't use Vault, but from the docs it looks like it's had that since the start? This is from 0.11.x which is the earliest documentation available: code:Current integration: https://developer.hashicorp.com/nomad/docs/integrations/vault-integration https://developer.hashicorp.com/nomad/docs/job-specification/template#vault-integration
|
|
#
?
Oct 12, 2023 08:25
|
|
|
I just assumed Nomad was using consul-template?
|
|
#
?
Oct 12, 2023 14:27
|
|
|
Nomad is completely adequate software, which is about the highest praise I can give to any of the modern container infra shitstack.
|
|
#
?
Oct 12, 2023 14:46
|
|
|
re: secrets management, there's an axiom I always fall back on: If you're writing a secret to a secrets database, and that secret isn't some kind of bootstrapping secret, you obviously aren't rotating that secret. Think about how to fix that situation (hint: have the thing doing your secrets rotation seed your secret too), and everything else about your secrets management strategy is going to fall into place around it without you even trying.
|
|
#
?
Oct 12, 2023 17:11
|
|
|
|
| # ? May 21, 2024 15:01 |
|
|
Nomad is reportedly on the lowest list of poo poo hashicorp works on and let’s face it. Hashicorp doesn’t have the best reputation lately. If you want a simple k8s then use k3s single node.
|
|
#
?
Oct 12, 2023 23:15
|
|
