Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Volguus posted:

github actions is free then? Even for private projects (which now are free, but weren't some time ago)?

Thanks to everyone for the recommendations, I'll have to pick my poison now. Worst comes to worst, it'll be jenkins, at least stackoverflow is full of pretty much anything when it comes to it.

quote:

GitHub Actions usage is free for both public repositories and self-hosted runners. For private repositories, each GitHub account receives a certain amount of free minutes and storage, depending on the product used with the account. Any usage beyond the included amounts is controlled by spending limits.

So yes. If you have a private repo, set up a self hosted agent.

Adbot
ADBOT LOVES YOU

Quebec Bagnet
Apr 28, 2009

mess with the honk
you get the bonk
Lipstick Apathy

New Yorp New Yorp posted:

So yes. If you have a private repo, set up a self hosted agent.

Gitlab is the same and can also be self-hosted for free, unlike GitHub. The free license is fine for what OP wants.

Volguus
Mar 3, 2009

chmods please posted:

Gitlab is the same and can also be self-hosted for free, unlike GitHub. The free license is fine for what OP wants.

Oh, wow, I had no idea that gitlab provides a build system in a self hosted environment. That would be probably perfect. I will definitely check it. And I can delay me having to learn anything about kubernetes for yet some more time. Thank you.

The Fool
Oct 16, 2003


Azure devops also provides cicd pipelines and free hosting fwiw

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

chmods please posted:

Gitlab is the same and can also be self-hosted for free, unlike GitHub. The free license is fine for what OP wants.

My point is that there's no reason to self host anything other than build agents unless you want the experience of maintaining your own on prem enterprise software.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Volguus posted:

Oh, wow, I had no idea that gitlab provides a build system in a self hosted environment. That would be probably perfect. I will definitely check it. And I can delay me having to learn anything about kubernetes for yet some more time. Thank you.

Kubernetes has nothing to do with build or continuous delivery. It's a platform for running containers.

Volguus
Mar 3, 2009

New Yorp New Yorp posted:

Kubernetes has nothing to do with build or continuous delivery. It's a platform for running containers.

Sure, but the other suggested build systems required it. That was all they were about. Gitlab (running it now) doesn't require it, but it does seem to support it. Now, if I have to figure out kubernetes ... eh, sure, fine. But if I don't have to, even better. From what I could gather in the last 24 hours is a good tool for a lot of use cases, but for a 1 man repository and build system, meh. You're right though that probably there is no real reason to host anything nowadays, as most services provide adequate support for most people's needs.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Volguus posted:

Sure, but the other suggested build systems required it.

One particular relatively esoteric build system (that frankly I'd never even heard of before) requires it. Most don't.

Kubernetes is a great tool but it introduces a ton of new concepts very rapidly and has a fairly steep learning curve. It's worth playing with at some point, but I'd leave it for after you're a bit further along than you are right now just because of how much there is to wrap your head around.

I'd never in a million years want to build and manage my own cluster outside of a trivial single-node minikube. Cloud Kubernetes services like AKS are a godsend.

The Fool posted:

Azure devops also provides cicd pipelines and free hosting fwiw

I love Azure DevOps but Microsoft's investing heavily in GitHub now with a clear if unspoken agenda that they are going to stop adding new features to Azure DevOps within the next few years and start herding everyone toward GitHub. Because of that I have a hard time recommending it to people who aren't already using it. I've heard that the bulk of the people working on Azure DevOps were shifted to GitHub a few years ago and Azure DevOps is largely running on a skeleton crew who are keeping the lights on.

I mean, look at the upcoming features: https://docs.microsoft.com/en-us/azure/devops/release-notes/features-timeline

Compare what they're planning on delivering for Q3 2021 to what was delivered several times a quarter back in, say, 2019. And notice that of the 5(!) things they're planning to deliver for the entire rest of 2021, two of them are better integration with GitHub. :hmm:

New Yorp New Yorp fucked around with this message at 07:47 on Sep 12, 2021

Mr Shiny Pants
Nov 12, 2012

Volguus posted:

I have a question: What build system should I use for my personal projects?

Background: I have a server in my basement that does not have enough VMs on it at the moment. I would really like to have a build system that would monitor some git repository, pull when changes are made to it and run a particular script and provide (or put somewhere) the resulting artifacts. This is just for personal projects. The less I'd have to gently caress around with it after setup the better.

The only build system I ever touched was Jenkins, and that was (and is) very briefly for little things. I don't have a problem with it, except that it seems to be a lot more than what I need. Is there a simpler one? I have a git repo on the network, now I'm using gitolite, but I wouldn't mind switching to another one if there's some tool out there that can combine git remote repo and CI and it would be relatively headache free for maintenance. I've heard Github itself does have some build system as well, I've never tried it. I usually only upload to github if I deem my project to be worthy of sharing, until then I just keep it on the local network. Though, now they do have free private repositories, but still ... why bother? But if their build system is worth learning, then it would be a compelling argument to just move to github.

What would you do for your own little projects?

Drone is nice. I use it in combination with Gitlab.

https://www.drone.io/

Docjowles
Apr 9, 2009

Out of curiosity, do many of you work at large shops where application developers have full admin access (not root, but basically allow *:*) to their applications' cloud accounts, with use of infrastructure-as-code tools being optional? My company is ostensibly trying to trying to modernize its applications and move large chunks of them to AWS. But we (the cross functional group trying to define our cloud strategy and deployment pipelines etc) are being met with total meltdowns and tantrums at every turn from developers who apparently cannot function without full console admin access and refuse to write Terraform/CloudFormation/CDK/any other proposed alternative. I don't think we've directly been compared to the gestapo yet, but like I would not be surprised at all to hear it dropped. I should also mention that this is not a startup. We have thousands of employees globally, do over a billion dollars in revenue annually, have been around for decades with a large data center footprint, are publicly traded with SOX and PCI compliance burdens, etc etc. It seems like basically a lot of the long-tenured developers were hoping to use the cloud as an end run around things like "security" and "caring about your infrastructure at all" and are extremely upset about being called on it. Unfortunately senior management is teetering somewhere between taking no stand, or caving to the the every-engineer-gets-admin-#yolo strategy.

I'm not the crazy one here, right? If you're building a large greenfield AWS deployment in 2021, your poo poo is going through a CI/CD pipeline, not clicking around the console like it's Windows NT? Trying to fight this battle for months has driven my morale to sub-zero.

The Iron Rose
May 12, 2012

:minnie: Cat Army :minnie:
you are entirely correct, and should be standing your ground in this fight

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Docjowles posted:

I'm not the crazy one here, right? If you're building a large greenfield AWS deployment in 2021, your poo poo is going through a CI/CD pipeline, not clicking around the console like it's Windows NT? Trying to fight this battle for months has driven my morale to sub-zero.

No, you're right. That cowboy poo poo shouldn't fly at any scope beyond "sandbox".

freeasinbeer
Mar 26, 2015

by Fluffdaddy
If your management are unsure, then uh, this can be a bad time going alone or screaming into the void. It can then turn on a dime when something like an auditor gets wind, and starts threatening to put your risk for tampering on your 10k.

There has also been a lot of bs/tech talks out of AWS for example talking about micro accounts, and giving full access to them, they are pushing that plus CDK as the answer to delegating access and segregation.

There are ways to enforce sanity, we’re using OPA + Atlantis to do it for terraform, and are now introducing the same guardrails to our cloud formation ecosystem.

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender
The way to frame it is responsibility for SEVs. If they want root, then they also take on responsibility if it goes sideways, and importantly you explicitly aren't responsible. If you're responsible, then it's your roof, your rules. They can't eat their cake and have it too.

12 rats tied together
Sep 7, 2006

also noteworthy that "devs have root/admin on production stuff" is an immediate failure on basically any type of worthwhile SOC compliance

FamDav
Mar 29, 2008

freeasinbeer posted:

If your management are unsure, then uh, this can be a bad time going alone or screaming into the void. It can then turn on a dime when something like an auditor gets wind, and starts threatening to put your risk for tampering on your 10k.

There has also been a lot of bs/tech talks out of AWS for example talking about micro accounts, and giving full access to them, they are pushing that plus CDK as the answer to delegating access and segregation.

There are ways to enforce sanity, we’re using OPA + Atlantis to do it for terraform, and are now introducing the same guardrails to our cloud formation ecosystem.

internal to aws, admin access to production is a carefully monitored if not completely verboten action. everything should be going through ci/cd pipelines leveraging cfn/cdk. you should be using many individual accounts to avoid the blast radius of even your cfn/cdk changes impacted multiple services, but that doesn't mean you should be giving anyone on the team full admin access save for "break glass/live preserver" scenarios.

Hadlock
Nov 9, 2004

1) no, developers should not have access to prod, period. You'll need to work with management and build rapport so that when prod goes down because snowflake star developer Dave couldn't get into his box to restart the service before he prod crashing bug that happens every 77 days, you don't get thrown under the bus. Management is going to look for a scapegoat after trusting you, and well bring padded clothes because you're getting thrown under the bus before you walk in the door Monday

2) at a previous company our CTO would log in and gently caress with poo poo, and we always knew it was him because prod files had his user listed as owner. The solution we came up with was to page on call whenever the CTO would log into a prod machine, because production would inevitably fail minutes or hours later

TL;DR do your best effort and document it in detail, but it's probably not worth losing your job trying to enforce something management isn't convinced of. You can't effect that kind of change without 100% buy in from senior management. Put the risk into dollars and customer aquisition cost and lost future revenue. That's the only language they speak

Good on you for recognizing this, you're better than 80% of employees out there already

Faith For Two
Aug 27, 2015
I’ve heard that you’re not supposed to add .o/.a/.d/.lib/.exe files in git. What are reasons for this, besides making your git history look clean? I guess storage space is one but maybe we have enough storage that we don’t care?

I wish I had something to say to my coworkers besides “wow what an **interesting** design decision. I’ve never worked on a git repo with no gitignore file before.”

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Faith For Two posted:

I’ve heard that you’re not supposed to add .o/.a/.d/.lib/.exe files in git. What are reasons for this, besides making your git history look clean? I guess storage space is one but maybe we have enough storage that we don’t care?

I wish I had something to say to my coworkers besides “wow what an **interesting** design decision. I’ve never worked on a git repo with no gitignore file before.”

Git stores files as chunks of text. It can't effectively do that for binaries so it has to store the complete version of every binary file. This makes the repo massive and eventually causes severe performance degredation. Even for local operations like checking out branches.

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender
Also there's no guaranteed correspondence between a binary and the source code (presumably in the same repo) that produced it. It's possible that foo.bin@HEAD was built with the surrounding source code @ HEAD... but how would you know unless the commit hash was somehow baked into the binary?

Vanadium
Jan 8, 2005

Who does have access to prod in y'all's setups if not the team developing the app? Do you have central ops teams that just set up the accounts/deployment systems for everybody else?

Edit: Like, I want to emphasize that I'm not doing some "lol who needs IaC/deploy automation" thing. Thankfully a central tooling team built the terraform and supporting code for our deployment pipelines, but we as the developer teams install it in our apps' accounts, and when the pipeline stops two thirds of the way through because the maintenance automation of yet another team got in the way somehow, it's probably me who grabs the audited near-admin credentials and untangles it. I don't think the tooling team would want to own that, but maybe that's just because we're doing the other stuff wrong?

Vanadium fucked around with this message at 16:28 on Sep 14, 2021

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Vanadium posted:

Who does have access to prod in y'all's setups if not the team developing the app? Do you have central ops teams that just set up the accounts/deployment systems for everybody else?

Edit: Like, I want to emphasize that I'm not doing some "lol who needs IaC/deploy automation" thing. Thankfully a central tooling team built the terraform and supporting code for our deployment pipelines, but we as the developer teams install it in our apps' accounts, and when the pipeline stops two thirds of the way through because the maintenance automation of yet another team got in the way somehow, it's probably me who grabs the audited near-admin credentials and untangles it. I don't think the tooling team would want to own that, but maybe that's just because we're doing the other stuff wrong?

"access" can mean many things. Who has full admin access and can do anything they want? No one. Who has access to poke around and look at non sensitive data? Lots of people.

In a health cross functional team there will be a support engineer or infrastructure engineer who has elevated access for the "Oh poo poo" moments, but the idea is that you should practically never have those because you developed a health self-healing platform backed by reliable, fast automation.

NihilCredo
Jun 6, 2011

iram omni possibili modo preme:
plus una illa te diffamabit, quam multæ virtutes commendabunt

Faith For Two posted:

I’ve heard that you’re not supposed to add .o/.a/.d/.lib/.exe files in git. What are reasons for this, besides making your git history look clean? I guess storage space is one but maybe we have enough storage that we don’t care?

Git is a source control mechanism. "Source" means that you store the fundamental files from which you can automatically produce artifacts, like native libraries and executables.

(And yes, it's designed primarily for text, but it's not wrong to save non-text files in git if they're original sources, e.g. graphic assets for a static website. If they're huge and impact performance, git-lfs exists for this purpose.)

Storing the .exe that is produced by compiling the .cpp files is redundant because you can already recreate the .exe by compiling those same .cpp files. It gives you no advantage and only causes the potential for chaos if the repo contains source code that produces a different .exe from the one that's stored. (For example, it could mean that the user can download a certain .exe, find a bug, report it, and the developer spends time debugging through source code to find a bug that doesn't exist.)

And if you need to store the .exe because you can't recreate the .exe starting from just the text files in your repo, that is a big loving problem because it means that your project isn't fully version-controlled. The git repository for a project should always enable you to fully recreate and modify the project from scratch by just cloning the repo and following the instructions in the README.

Vanadium
Jan 8, 2005

New Yorp New Yorp posted:

"access" can mean many things. Who has full admin access and can do anything they want? No one. Who has access to poke around and look at non sensitive data? Lots of people.

In a health cross functional team there will be a support engineer or infrastructure engineer who has elevated access for the "Oh poo poo" moments, but the idea is that you should practically never have those because you developed a health self-healing platform backed by reliable, fast automation.

This basically checks out, but I guess we also have access to grant full non-root admin access subject to review by ourselves and presumably heavy auditing, does that still count? I guess my team isn't so much cross functional as "you code it, you own it". And yeah, every time I actually have to touch an account outside of mayyybe looking at non-centralized logs with a super restricted role, it's Not Great, but also I guess not a huge priority to get out of. We get funny looks if we spend time on deployment tooling kinda thing because we're not that team, and that team is juggling a lot of other high priority responsibilities. welp.

... back in the day, when we all lived out of a shared account, our ops team was very conservative giving idiot newbies like me IAM permissions, but the consequence was that I couldn't use terraform to spin up my thing or create roles for my thing to use because we had no central deployment platform or anything. After a bunch of back and forth between teams, a director eventually decreed that all team leads get *:* and then my team lead was like "I don't have time for AWS stuff, you do it" and gave me *:*, and then everything was kind of scary for a long while until we figured out that we could have more than one account...

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender

NihilCredo posted:

Git is a source control mechanism. "Source" means that you store the fundamental files from which you can automatically produce artifacts, like native libraries and executables.
I agree with everything you said; but it sounds like this team probably knows they're abusing Git, so they won't listen to appeals to principles. And I suspect they're doing it because Git can be easily abused as a convenient way for transferring files around. An age ago, our jerry-rigged deployment system was "log into prod and do a 'git pull origin master'", because we didn't have the time/knowhow to properly build, bundle, and Chef/Puppet/Ansible the code into production. It might be that in this team's case, using Git to store binaries is a symptom of a very poor deployment system. If that's the case, addressing that first might be a way to convince them to not abuse Git for that purpose.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

minato posted:

I agree with everything you said; but it sounds like this team probably knows they're abusing Git, so they won't listen to appeals to principles. And I suspect they're doing it because Git can be easily abused as a convenient way for transferring files around. An age ago, our jerry-rigged deployment system was "log into prod and do a 'git pull origin master'", because we didn't have the time/knowhow to properly build, bundle, and Chef/Puppet/Ansible the code into production. It might be that in this team's case, using Git to store binaries is a symptom of a very poor deployment system. If that's the case, addressing that first might be a way to convince them to not abuse Git for that purpose.

To be fair I once proposed doing something like that with git lfs for a team that stored literally gigabytes of static content in source control and had a nightmarish process for rolling out content changes. Didn't end up doing it but it absolutely would have been an improvement for them. Getting them to use a proper cdn would have been the best solution but they absolutely would not consider changing their work flow, only the method of rolling the changes out.

Docjowles
Apr 9, 2009

freeasinbeer posted:

If your management are unsure, then uh, this can be a bad time going alone or screaming into the void. It can then turn on a dime when something like an auditor gets wind, and starts threatening to put your risk for tampering on your 10k.

There has also been a lot of bs/tech talks out of AWS for example talking about micro accounts, and giving full access to them, they are pushing that plus CDK as the answer to delegating access and segregation.

There are ways to enforce sanity, we’re using OPA + Atlantis to do it for terraform, and are now introducing the same guardrails to our cloud formation ecosystem.

Yeah we give each team their own set of AWS accounts, and are attempting to use Terraform run through Atlantis but this is apparently not good enough (read: people don't want to learn Terraform when they could just mash buttons in the console cause GOTTA GO FAST)

12 rats tied together posted:

also noteworthy that "devs have root/admin on production stuff" is an immediate failure on basically any type of worthwhile SOC compliance

Amusingly we just had a call with the compliance department and a third party auditor yesterday. And as we walked through the direction we're being forced down for operations in the cloud they were clearly unimpressed, lol. So I do think this may end up whipping back around to sanity at some point in the future when someone gets told we can't pass an audit and they won't sign off on our financials. But the fact that we are having to fight tooth and nail for this is an enormous red flag and has me looking to get out.

Hadlock posted:

1) no, developers should not have access to prod, period. You'll need to work with management and build rapport so that when prod goes down because snowflake star developer Dave couldn't get into his box to restart the service before he prod crashing bug that happens every 77 days, you don't get thrown under the bus. Management is going to look for a scapegoat after trusting you, and well bring padded clothes because you're getting thrown under the bus before you walk in the door Monday

minato posted:

The way to frame it is responsibility for SEVs. If they want root, then they also take on responsibility if it goes sideways, and importantly you explicitly aren't responsible. If you're responsible, then it's your roof, your rules. They can't eat their cake and have it too.

Also this. There's lip service to the idea of the application teams having full responsibility for outages and problems caused by their elevated access. But I have complete faith that if there's a catastrophic outage or we're on the news/in court because some dipshit put PII in an RDS instance on a public IP with no authentication or whatever, someone's going to be screaming "HOW COULD OPS AND SECURITY ALLOW THIS TO HAPPEN YOU HAD ONE JOB". It's also not a great look for your resume to have been working on a company's cloud deployment during a time they made headlines for loving up in the cloud.

Yeah the more I sit here and think through the way senior management is driving the bus, the more I want to just bail out before it crashes.

Hadlock
Nov 9, 2004

Docjowles posted:

we just had a call with the compliance department and a third party auditor yesterday. And as we walked through the direction we're being forced down for operations in the cloud they were clearly unimpressed, lol. So I do think this may end up whipping back around to sanity at some point in the future when someone gets told we can't pass an audit and they won't sign off on our financials.

Yeah if you're looking to go IPO and going through the motions of doing an initial security audit and management rolls their eyes at the thought of seperating developers from prod, well, you might as well buy out your options and move somewhere else, that IPO is not happening in the next two years for sure. The fact that you're recognizing this as a major red flag is probably a good thing.

Also, everyone is hiring like crazy right now, you can probably jump ship for at least a 25% raise, so that's something.

12 rats tied together
Sep 7, 2006

as time goes on i am more and more convinced that "development teams own their infrastructure and it is deployed through code they write" is a naive fantasy

even if you could will the aggregate motivation and care required for 100 non-infrastructure developers to collaborate on terraform into existence, it would be a waste. you'd consume an irresponsible amount of developer-effort-money-time "organizationally learning" a problem space that can be adequately covered by like a 1:100 ratio infrastructure SME team

NihilCredo
Jun 6, 2011

iram omni possibili modo preme:
plus una illa te diffamabit, quam multæ virtutes commendabunt

minato posted:

I agree with everything you said; but it sounds like this team probably knows they're abusing Git, so they won't listen to appeals to principles. And I suspect they're doing it because Git can be easily abused as a convenient way for transferring files around. An age ago, our jerry-rigged deployment system was "log into prod and do a 'git pull origin master'", because we didn't have the time/knowhow to properly build, bundle, and Chef/Puppet/Ansible the code into production. It might be that in this team's case, using Git to store binaries is a symptom of a very poor deployment system. If that's the case, addressing that first might be a way to convince them to not abuse Git for that purpose.

Honestly, using git as a poor man's SFTP is sloppy but far, far less of an issue compared to the lack of a standardized build process.

If the scenario were as you describe it, I would propose - as a compromise first step - to keep the git pull "abuse" and just add an actual build server. An isolated machine with a script running that pulls every minute, if it finds a new commit it compiles the project, commits it to ./bin, tags it as ci-release-(n+1) and pushes it. Then change the deployment script so it only pulls those tags.

Now you have a semi-decent guarantee that the binaries match the source, with little or no impact to the existing workflow.

Methanar
Sep 26, 2013

by the sex ghost

12 rats tied together posted:

as time goes on i am more and more convinced that "development teams own their infrastructure and it is deployed through code they write" is a naive fantasy

even if you could will the aggregate motivation and care required for 100 non-infrastructure developers to collaborate on terraform into existence, it would be a waste. you'd consume an irresponsible amount of developer-effort-money-time "organizationally learning" a problem space that can be adequately covered by like a 1:100 ratio infrastructure SME team

Hard agree.

Methanar posted:

I spend a significant chunk of my time moving devs away from owning their own infra precisely because they don't have the time to do it properly themselves. Even stuff like using a standardized instance type so we can properly spec and plan for RI pricing and prepurchasing instances up front is a major costs savings that we are only achieving by consolidating that ownership over to ops.

Spot instances, autoscaling, reducing cross AZ traffic, using right-sized EBS volumes in terms of IOPs are other big ones too. What happens when you give devs the responsibility to self service all these things in a fast growing company they just massively oversize everything so they don't need to think about it.


Methanar posted:

I don't know what scaling operations means if not to reduce the cost it takes to run the platform. I've easily reduced annual spend by my salary in each of the last 2 years.

I don't own anybody's app. I just am responsible for providing kubernetes, the ecosystem around kubernetes, and making sure everybody is using kubernetes properly. Kubernetes is the platform and mechanism through which ops is taking ownership of the infra what was previously yolo mode.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
The regulatory line that causes people to demand devs have no prod access is from SOX requiring separation of roles and duties among persons. Obviously this is true, but the conservative interpretation is "devs don't get any access to prod" while I've seen SOX organizations give access with proper oversight. I don't want to be the person being virtual keyboard for someone else, so with tools like Gravitational Teleport and ssh certificates I can authorize users access to prod during emergencies and have it be well documented.

As long as you have _accountability_ and auditability of your systems with a well-documented and enforced defense in depth architecture for authz/n you're probably alright. Been through various SOC and SOX processes before across different verticals.

I will also note that a big reason for companies moving to K8S is because they want to avoid giving developers any cloud credentials (see: all those fun Github AWS key leaks) and keep them focused upon application development with strong boundaries (a healthy relationship has strong boundaries and trust, right?).

Vanadium
Jan 8, 2005

Coming back into the thread because today the platform and tooling people floated the proposal that they would stop owning the ssl certs and instead the development teams would deploy their code along with a config file that tells an utility which certificate to pull in. If you as a developer don't update the config file and do a deploy in the 60 day window before expiration, you get paged at some point. This is a probably going to be fun because there are so many deployment targets that we'll probably always be in someone's 60 day window and our deployments need a frustrating amount of developer attention currently (including, afaict, ssh'ing into prod to resolve/investigate deploy failures), but also in at least one case we need a certificate updated on hosts that are completely distinct from where the team's service gets deployed to, so I have no idea what the hope is for them. Developers have raised objections along the lines of "I don't actually know what a certificate is and I suspect our new hires aren't getting trained about that either".

I definitely see a few of the constraints that lead towards this looking like the best possible solution at least short- to medium term, but the overall situation is also instilling a rather powerful urge to run, preferable towards a place where I as a developer am not allowed to touch prod, after all.

If my pal in the aforementioned tooling teams is reading this: i feel i like i ought to buy you a drink of some sort.

12 rats tied together
Sep 7, 2006

necrobobsledder posted:

The regulatory line that causes people to demand devs have no prod access is from SOX requiring separation of roles and duties among persons. Obviously this is true, but the conservative interpretation is "devs don't get any access to prod" while I've seen SOX organizations give access with proper oversight.
Certainly some aspect of this is true, but it also depends on the context, since not every business is bound by or particularly cares about SOX.

Most of my experience in this space is at a data science consulting firm and our SOC compliance was geared toward convincing large companies and various government agencies that it was OK to give us hundreds of millions of social security numbers, census data, banking data, things of this nature.

In this context we took it the opposite way: technically nobody has access to "prod", not even ops, without going through some serious and very manual escalation workflows.

necrobobsledder posted:

I will also note that a big reason for companies moving to K8S is because they want to avoid giving developers any cloud credentials (see: all those fun Github AWS key leaks) and keep them focused upon application development with strong boundaries (a healthy relationship has strong boundaries and trust, right?).
I agree with this but I also want to point out that it's wrong, and is one item on a very large list of wrong and stupid reasons why companies are moving to k8s.

We were supposed to have been doing this all the time anyway. Nobody has ever needed to give developers keys, companies who think the k8s api will solve for their lack of process are just moving all the bad into something they don't understand enough to notice the bad in.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
I support containerization moreso than I support K8S mostly because it's a cleaner separation of concerns between infrastructure engineering and application engineering for most application development environments at scale. Of course it's not applicable for every situation, but anything that helps people focus more upon things they're actually experts at / enjoy and can minimize their deficiencies / unenjoyment is good technology IMO.

A lot of companies try to substitute technologies for having garbage processes to begin with, but I'm also quite familiar with garbage, over engineered processes as well which is quite possible without even mentioning containers or K8S. At least with K8S it's a better documented mess that can be pushed to a group of people that are better at process standardization. The blast radius of credential compromises with AWS IAM leaks is probably much worse than a developer with minimal production access having keys for their kubectl setup compromised.

12 rats tied together
Sep 7, 2006

Totally agreed. A container as a mechanism for describing effectively a "linux process namespace" is a great piece of technology.

A container that is treated organizationally as "a vm but with helm" is bad. I've had to field requests for a "m5.large container" which IMO is the canary in the coal mine of resume driven development.

It is my experience that almost always the container comes into play when there exists a team in between infrastructure SME and feature development. Often in these cases k8s becomes a replacement for linux, as if we can solve legacy application problems by copying the code into a .net core container and pushing to k8s, which is its own special kind of toxic.

12 rats tied together fucked around with this message at 17:49 on Sep 17, 2021

Hadlock
Nov 9, 2004

Terraform versioning modules vs monorepo,

and go

The Iron Rose
May 12, 2012

:minnie: Cat Army :minnie:

Hadlock posted:

Terraform versioning modules vs monorepo,

and go

Not entirely sure what you mean by versioning modules. I usually have the calling module handle the version, Not the child module.

Monorepo is fine so long as you don’t have the same state file for everything. We have a big monorepo at work and it takes like 8 minutes just to run the terraform plan. I’m fine with everything living in the same repo. Less so with everything sharing state.

barkbell
Apr 14, 2006

woof
What's a good way for having parity with your pipeline and local development?

Like, I don't really want my Jenkins steps to include a lot of docker options right? Those should be put somewhere else I assume, because then if I want to emulate what the pipeline does I can't do that locally. Do I run kubernetes localy?

I'm an idiot and this is probably a solved problem most places I just don't know where to start researching

Adbot
ADBOT LOVES YOU

Methanar
Sep 26, 2013

by the sex ghost

Hadlock posted:

Terraform versioning modules vs monorepo,

and go

If the modules are only used by one well defined ish team just do monorepo

At the very least don't let your versioned modules themselves have versioned modules they pull. It becomes a major pain in the rear end to update anything in this case.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply