Space Whale posted:

So the "branch off of master, merge all into a dev branch; merge that dev branch into master to deploy" pattern has my architect a bit antsy, since with TFS it was hell. I've also seen at least one nasty git merge, but we were having spaghetti merges that looked like a cladogram of the protists.

...why not just use that dev as your master? What would make it all go pear shaped? I know that the project files can get a bit messy, but seriously, why?

Space Whale posted:

That raises another question I've gotten.

Let's say that there are breaking changes made by work being done in the API or architecture of the project, or whatever. These changes all need to be shared and the breaking bits fixed. Would pushing that to master early then doing the rebase/merge from master onto their branches be the best way to do this?

smackfu posted:

Aye that sounds good. Since our full test suite takes 15 minutes to run, even with our pretty small team we run into that conflict issue and it does waste time.

It's kind of a bummer... the BitBucket feature that I am piloting does the build *pre-merge* which is kind of annoying because it still means the merged code can break the master build. Your system seems better.

minato posted:

I think it comes down to splitting the update into 2 phases:
1) Ensure the desired state description and necessary assets are on the machine.
2) Initiate a server-local process to update the desired state from those local assets.

1 can be retried until successful with no side effects, which mitigates the issue of poor or intermittent connectivity. And 2 is likely to succeed because all the assets are in place and no connectivity is required.

Vulture Culture posted:

This could really be as simple as a systemd unit that downloads a compose file and runs it, if you're capable of running your reporting out of band

Erwin posted:

Do you have control over the connection and purposefully bring it down to limit costs, or is it a random as-available thing? As much as I wouldn't recommend Chef to anyone these days, this sounds like something Chef is more suited for than Ansible, if you wanted to use an existing CM tool. Since Chef runs as an agent in a 'pull' model, each client would hit the Chef server whenever they had connectivity to grab any new configuration. Looks like the Docker cookbook can manage containers as well, but I think I'd separate system configuration and container orchestration and update a compose file like VC said.

If you have the budget for it, I know first-hand that DC/OS runs on cruise ships with sporadic satellite connections. I have strong opinions on DC/OS and simple container orchestration wouldn't be a use case I'd pick it for, but I know it's successfully solving the problem you are trying to solve.

Hadlock posted:

Check out CoreOS and their cloud config files

Basically you pull down a new cloud config file, then reboot the server. The server executes the cloud config file (run these containers with these arguments) and away you go. If the cloud config file fails, it boots from the previous safe config. Maybe setup a cron job to do updates at predicted connectivity periods.

This sort of follows the telcom/networking model of "new update, but fail back to the previous version if it doesn't work"

CoreOS uses Omaha protocol to poll for updates, but that is an entire other rabbit hole.

TL;DR CoreOS was designed from the ground up to do exactly what you plan on doing

New Yorp New Yorp posted:

It's even worse when a company just renames their operations team to "devops" and puts a bunch of people with somewhere between "zero" and "minimal" prior development experience in charge of writing software.

NihilCredo posted:

I'm looking into a storage abstraction for a product so it can run with no code changes in anything from a piddly under-the-table VM with a plain HDD to a MS/AWS/GC environment with that vendor's blob storage. Ideally, it would also handle a "I have a poor man's datacenter, N physical machines running orchestrated containers and some network storage (either a NAS or even each one with their HDD), please replicate my data as much as you are able without giving it to those icky cloud vendors" scenario, which I'm really hoping to avoid but cannot dismiss out-of-hand .

Is Ceph what I'm looking for or is there a less overkill solution? I've seen Storidge advertised around which seems simpler, but also very unproven.

Nomnom Cookie posted:

Try that and your system will end up not working with real s3 unless the emulator vigorously enforces metadata inconsistency. S3 provides fewer guarantees than almost any storage system and stale reads are common

Walked posted:

It's been a while since I did a survey of the field for CI/CD systems; just changed jobs and get to do it again.

Any new players in the last 1-2 years worth checking out? We were on CircleCI at my last place and it was fine; dont mind using it again but want to be sure I'm not missing anything making waves more recently.

Walked posted:

Drone always stuck out to me as a sweet option; but never heard anyone else using it.

I’ll give it another peek

Newf posted:

Just checking: is it a bad practice for mydomain.com/.env.production.local to happily serve up a file with, eg, admin database credentials?

edit: asking for a friend

Boz0r posted:

What is it called in Azure DevOps when committed code gets rejected if it doesn't build and pass all tests? People from our team break our pipelines all the time and I'm sick of it.

Gangsta Lean posted:

SAFe loving owns. Previous job flew or bought Amtrak for engineers, engineering managers, product managers, executives, *scrum masters*, etc from remote offices all across the country, even Hawaii, to a city in the northeastern US for 2 fun-filled days of sitting in a room together discussing tickets. Huge breakfast, huge lunch like 3 hours later, then after the afternoon session was over they leave you to find your own way back to a hotel in the middle of an isolated office park with nothing within walking distance except more office buildings. All the non-US-based contractors (there were a ton, mostly in eastern Europe) were not invited, they dialed in via Zoom just like they did every work day.

Executive management floated around and sat in random sessions for a few minutes throughout they day, interjecting opinions that everyone promptly ignored when they left. At the end of the second day everyone gathered to hear each team commit their soul to some unrealistic goal, lots of head nodding all around, but failing to meet a commitment never resulted in any consequences that I ever saw.

The best part about SAFe is, when management tells you to do something counter to SAFe (it happens a lot), the company buy-in is so hard that all you have to do is point out how contradictory what they’re asking is to the SAFe philosophy. It’s an excuse for following the process instead of doing the work that actually needs to be done. The product is now the process, not the software you’re actually delivering to customers.

NihilCredo posted:

We're using Gitlab and are quite happy with it as well.

Sometimes it's got minor bugs that have been left open for 2+ years in favor of adding more enterprise paid features, which I can't really blame them for. None of those have been show-stoppers, just stuff like the build cache not triggering and slowing down builds by a few minutes.

I might consider Gitea if I did not need a built-in CI/CD system or built-in package manager, and/or if I didn't have a beefy server to host it on. Gitlab is a massive resource hog, while Gitea runs on a Pi and feels blazing fast at all times.

Then again, Gitlab is an enterprise product with all that it entails, e.g. I've never had a single issue running a plain `gitlab backup create && apt-get upgrade` after a new release; whereas Gitea is an open-source project that isn't even dogfooding itself yet (is code hosted on Github).

edit: Gitea apparently supports git mirroring (while it's a paid feature in Gitlab) so you can maybe install both with mirrored repos and get a feel for which one you like better.

New Yorp New Yorp posted:

Needs more context. What clouds are you using? Are you targeting Kubernetes? What's your technology stack? What are you using for CI? What automation do you already have in place? Are you the only person who will be supporting this, or will it be a team effort?

Hadlock posted:

Terraform is still good, better since v0.11 or 0.12 when they sort of committed (briefly) to a 1.0 spec

Terraform is better if you're doing Career-Driven Development as it's a transferable skill and most everybody interviewing you will nod their head when those words come out of your mouth

edit: kind of wondering what happened to terraform 1.0? I guess they're afraid that if they publish a stable spec, the community will fork it and add all the good features people have been wanting for years and leave them in the dust? Terraform is like this close >.< to being truly great but has a bunch of weird gotchas related to opinionated decisions the community has zero control over

Vulture Culture posted:

One person on one of our teams did a Terraform configuration generator and it took them like three days to do it. This should not be difficult for a company whose Sentinel engineering team is bigger than most software companies

the talent deficit posted:

terraform consumes so much time and attention at every place i've been that used it that i'm convinced it's a scam to ensure full employment of programmers who don't want to program

Methanar posted:

fletcher posted:

For us terraform has been more of a "set it and forget it" type of experience, it's been great

Methanar posted:

Separate things by area of concern. Shared infra like vpc subnet network stuff should be independent.

Then do one terraform project per stack of whatever you're doing.

Tag management is another perfect example of something else that is shared and should be broken down into a module and attached to other terraform projects. We have an org wide set of tf modules with all the environments defined in a big struct full of key values for what cost accounting tags to use. What the chef server urls are. What the aws region is. That sort of thing. Just one place to update and then every thing else inherits.

Volguus posted:

I have a question: What build system should I use for my personal projects?

Background: I have a server in my basement that does not have enough VMs on it at the moment. I would really like to have a build system that would monitor some git repository, pull when changes are made to it and run a particular script and provide (or put somewhere) the resulting artifacts. This is just for personal projects. The less I'd have to gently caress around with it after setup the better.

The only build system I ever touched was Jenkins, and that was (and is) very briefly for little things. I don't have a problem with it, except that it seems to be a lot more than what I need. Is there a simpler one? I have a git repo on the network, now I'm using gitolite, but I wouldn't mind switching to another one if there's some tool out there that can combine git remote repo and CI and it would be relatively headache free for maintenance. I've heard Github itself does have some build system as well, I've never tried it. I usually only upload to github if I deem my project to be worthy of sharing, until then I just keep it on the local network. Though, now they do have free private repositories, but still ... why bother? But if their build system is worth learning, then it would be a compelling argument to just move to github.

What would you do for your own little projects?

Volguus posted:

I run/work in linux. A windows VM on my machine would have the same issues as the one running in proxmox: need to be replaced every 70+ days. What I was looking for was an automated way to do that (replace that VM). The obviously easy thing is to just use some key and set it to never expire and just move on with life. But I thought I may as well explore other possibilities before I'd go there. The scripts that I have make me having a working VM capable of building stuff in around 30 minutes to 1 hour, but they require me to click buttons. It's not a big deal once every 70+ days.

Methanar posted:

In the last 48 hours I've turned off 3500 CPU cores worth of poo poo that was completely unnecessary. And I've got at least another 3000 I can turn off in the next week.

Absolutely ridiculous how wasteful and careless people are.

The Iron Rose posted:

I could but the fact that customers are banging on the drums here is a limiting factor. There’s lots of approaches here with service meshes though. There was a whole ‘nother solution I briefly entertained of using canary traffic splitting to do the same thing before I realized I was trying to fit a square into a round jole.

Which frankly is what’s happening in general, and there’s no way I’d want to actually implement the lua plugin.

Anyways this thread remains the best grey tech thread, I continue to learn a phenomenal amount from reading what folks post here. Never even heard of CQRS before and it’s fascinating. And 2000s era tech read replicas may be, but it’s quite new and exciting round these here parts.

E: With regards to splitting traffic between read and write services, it’s really a terrible idea and the difficulty of implementation reflects that. While a fun afternoon’s research and testing, this is not the model we’re going to use. With the requirement to split traffic across multiple services with session affinity gone, the problem set becomes much simpler. More importantly, the service becomes easier to comprehend and maintain, we reduce reliability and quality problems, and we preserve flexibility in how we design our application going forwards.