Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
syphon
Jan 1, 2001
It's definitely a trade-off. On one hand, having the deployment configs generated with the build give you an atomic "package" that's easily reproducible. On the other, it's not very agile, and if you need to deploy that build to a new environment, you're screwed. Alternately, having your deployment configs separate from your build create more flexibility, but becomes another config setting you're dependent on (it becomes more difficult to wholly reproduce a 'deployment', because a change could have come from either the build or the config side of things).

You can use Source Control to try to mitigate this problem. Things like Git's 'tag' or Perforce 'label' (I'm sure TFS has to have an equiv) can help, but I've never found a rock-solid solution to this problem when you start deploying your app to many different 'environments'.

Adbot
ADBOT LOVES YOU

syphon
Jan 1, 2001
Tools like Chef have done a really good job of mitigating this problem. Your cookbooks have 'defaults' which can be overridden per environment. This answers minato's stated problem of "configuration drift" across environments. Then, you can enforce versioning of your cookbooks in order to create reproducible deployments.

Managing the mapping of App Version to Cookbook Version is a bit of a pain, but I think the benefits outweigh the costs.

syphon
Jan 1, 2001
Ah I briefly met him at this year's Chefconf. Seemed like a really smart guy.

syphon
Jan 1, 2001
The enterprise version of Jenkins has a Templates Plugin for creating similar jobs from a template. That doesn't solve the "elephants all the way down" problem you describe (I love that term, btw) but it might help with this one specific instance.

syphon
Jan 1, 2001
This is the book I've seen most people recommend as the place to get started with CI/CD. It's commonly referred to as the "Black book" - http://www.amazon.com/Continuous-Delivery-Deployment-Automation-Addison-Wesley/dp/0321601912

As far as things to do first, I'd say set up a CI server like Jenkins. It'll build .Net solutions just fine (although other for-pay tools like TeamCity do it a bit better). Get it pulling from your SVN server, running msbuild with every commit, then publishing a versioned artifact to your file share somewhere.

From there you can work on automating your deploy process, then enhancing your branching strategy. It may be worth going with Github for source control (it doesn't have to be publicly available), as Github has tremendous workflows for easy branching a code-reviews and whatnot.

syphon
Jan 1, 2001
Yeah you can use your own hosted version of Github (GHE) but it costs money and resources to stand-up/support so the ROI drops dramatically. Given those options I'd stick with SVN for now. No reason to change EVERYTHING at first!

I'd probably tackle your challenges in this order:
--1) Set up CI builds with something like Jenkins
--2) Automate your deployments
--3) Create branch model that supports CI/CD. A Trunk or 'Mainline' setup is pretty common for this, so you're not too far off base.
--4) Automate your testing (this is a huge endeavor so don't expect to get this one done easily :))
--5) Tie it all together in a Continuous Delivery Pipeline

One of the biggest challenges I see from moving teams into a CI/CD model is the concept of "Every check-in must be capable of shipping all the way to release". If they've been doing it for a long time, people get way too used to the concept of "I can always check-in a fix later" and break the build or commit their half-written code. The more devs you have working in this mindset, the longer your build/deploy/tests will be broken, people will be blocked, and you're not releasing software. The idea is to set up your branch plan so that it allows people to commit frequently, but ALSO not commit junk to the Trunk branch and break everyone else (Github is really good at this by default).

syphon fucked around with this message at 23:18 on Jun 2, 2015

syphon
Jan 1, 2001

Stoph posted:

I feel like proper code quality and CI is an insurmountable task until we switch to a pull request based workflow.
I don't really know SVN at all, but I've used both Perforce and Github. Perforce has a code-review system that relies on "Shelved Changelists", is there anything similar for SVN?

syphon
Jan 1, 2001

Ithaqua posted:

The other big challenge is to get people to start using feature flags and short-lived dev branches so you can ship your code even if a feature is half-completed. The killer is usually database stuff -- it's hard to get into the mindset of never (rarely) introducing breaking database schema changes.
Feature flags are great, but as the team/product gets larger they can turn into their own nightmare. For example, if you have 20 different devs contributing to the same product and they all put their code behind feature flags, that's 20^2 permutations of the app that should be tested (each feature should be tested against every possible permutation of every OTHER feature flag). What if another team has to roll back their changes or turn their feature off? Are you sure your code works reliable with features A B and C turned on but features X Y and Z turned off?

syphon
Jan 1, 2001
Yeah my example was certainly a bit exaggerated... but my company has a pretty large monolithic app where it's not unheard of to have 10+ feature flags spanning multiple releases all going at once. I'm willing to admit we're a bit of an edge case there. :)

syphon
Jan 1, 2001

bgreman posted:

Do you work at my office? We've been revamping our CI/CD scheme over the last six months, migrating from many long-lived "feature" branches to one trunk branch + feature flags, and the pushback has been incredible, particularly with database stuff.
I'm not surprised by this at all. It seems to me that DB changes (schema mostly) are the hardest to make forward/backward compatible (and thus able to be put behind a feature flag). FWIW though, it was a HUGE win when my company was finally able to accomplish this! Keep fighting the good fight!

Adbot
ADBOT LOVES YOU

syphon
Jan 1, 2001
The idea is to treat all of your configs as code (which basically means put them in source control and run them through whatever "build/deploy/test" processes are applicable). There are various "Configuration Management" tools (chef, puppet, ansible, salt) that support and encourage these practices.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply