Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
duck monster
Dec 15, 2004

EAT THE EGGS RICOLA posted:

Docker has had a few crazy bugs recently that would make it insane to use in production, hasn't it?

Yeah I was working at a govt department with a *lot* of Django apps and we tried to set up a hg -> jenkins -> docker -> test -> deploy kind of chain but docker was just too fragile to really be worth it. Its awesome in theory but in practice its just not solid enough to do what we wanted it to, which ultimately was to build an inhouse heroku type set up so the various coders in the sub departments could deploy their poo poo without being given the keys to the castle on out various servers. Plus as far as security goes its actually *less* secure than a chroot jail.

Theres also that COREOS thing that your supposed to deploy it onto but we found etcd to be completely flakey.

Adbot
ADBOT LOVES YOU

duck monster
Dec 15, 2004

NovemberMike posted:

What about Saltstack? I've been playing around with it and it seems nice ,anyone have real opinions?

We ran a very large government department with it. Science clusters, windows servers, various linux boxes, virtual hosts and servers, the lot. Its very nice. Like all of these things, theres a bit of a learning curve, but honestly I found it much easier than puppet

duck monster
Dec 15, 2004

So recently I started a new job that partially involves inhereting a giant Kubernetes cluster on DigitalOcean. I've never used Kubernetes so its all a massive learning curve.

This moring I got into the office and realised the entire cluster down with all the pods in "Pending" mode (Including about a bazillion cronjob containers that seemed to be piling up).

It would seem at some point in the night for reasons I'm completely unsure of the whole drat thing was reset causing it to reissue a whole bunch of nodes which where in an unlabeled state.
So after labelling them, it all came back up, although I had to delete the node spec for the cronjobs because there where literally hundreds of the bloody things trying to be created. Followed by a slow recycling of nodes to get the drat things to exit the "Terminating" state.

Massive and disrupive pain in the arse.

Is there a way to tell Kubernetes how to label nodes after a rebuild? Beause this *sucks*

duck monster
Dec 15, 2004

my homie dhall posted:

The process you should be looking at is kubelet. Looks like you can modify the kubelet config to have the kubelet come up with whatever node labels you want.

Thats not a thing you can do with Digitaloceans kubernetes system.

duck monster
Dec 15, 2004

necrobobsledder posted:

According to this Github issue there's a CLI tool you can use to set the node labels https://github.com/digitalocean/DOKS/issues/3

Yeah i ended up figuring out that digital oceans cli can do it too.,

doctl kubernetes cluster node-pool update <cluster name> <nodepool name> --label <key>=<value>

I probably have to learn about how the taints and affinities and poo poo works.
("Hey love , guess what I did at work today? I put labels on taints! Do you have an affinity for taint? Better label that too! I'll go drain the pool.").

duck monster
Dec 15, 2004

Woof Blitzer posted:

What kind of sick individual invented YAML anyways

Yet Another Malajusted Lout.

Adbot
ADBOT LOVES YOU

duck monster
Dec 15, 2004

This is fun. Deploy script that uses IMAGEVERSION var in .deploy to drive a few things in k8

code:
source .deploy
export API_IMAGE=registry.digitalocean.com/<stuff goes here>:$IMAGEVERSION
doctl registry login
docker build -t <stuff here>:$IMAGEVERSION .
docker tag <stuff here>i:$IMAGEVERSION $API_IMAGE
docker push $API_IMAGE
pushd <k8s dir>
./yg e -i '.spec.template.spec.containers[0].image = strenv(API_IMAGE)' api/api-deployment.yaml         #Update the k8s yaml
kubectl replace -f api/api-deployment.yaml
popd
Theres a bit more to it thats stuff that'd get me in trouble to reveal. But thats a nifty little script that we put on a git hook to a deploy branch and magic! Instant deployments. Just update the IMAGEVERSION .deploy in the git repo, push to del, and your good to go.

Next step is to get some CI in on it (Probably jenkins) to run the tests and make sure we're not pushing hot garbage. I think. If the boss will let me

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply