|
jaegerx posted:I do openshift if anyone wants help with that. I'm sorry
|
#
?
Jan 28, 2018 06:20
|
|
|
|
| # ? May 15, 2024 10:33 |
|
|
Virigoth posted:Do you have a debrief of what went wrong? There is so much kubernetes running on AWS that they made a service specifically to help more people use it on AWS. Our TAMs were happy they pushed the kubernetes service so fast because it was blowing up their time from enterprises. i don't think k8s is bad or anything i just think it's a big change from running your own data center. if they'd setup a k8s cluster onsite and moved what was easy to move to k8s they would have built valuable experience in k8s that would have helped moving the hard to move parts. similarly, if they'd moved to aws via using managed services like rds/elasticache/emr where appropriate and simple autoscale groups for their application tier i think they would have had issues but would have had a better chance of success. there was no chance they were going to do the whole data center to k8s on aws migration tho
|
|
#
?
Jan 29, 2018 05:40
|
|
|
Punkbob posted:Edit: the team that uses kops does do the terraform export, but it’s one of the things that I don’t understand why they do it or fight so hard with it besides just really liking terraform. I've agreed to do terraform export for my items; I don't like it a whole whole lot, but on the other hand, it's good to have your state documented in code somewhere that's readable by a third party tool. We applied for the EKS managed kubernetes aws beta, haven't heard anything back from them yet. We have one guy using fargate to vastly simplify our QA/selenium stuff but looking at the numbers it's like 2x cost of running it in self-managed k8s. We're hoping to see EKS pricing and hope it's not insane like fargate. Punkbob posted:I’d switch from kube-lego to cert-manager, it’s by the same folks but is a better spin on what kube-lego does and has features like using dns verification so you don’t have to expose everything to the world. Thanks for the suggestion I will definitely check it out. The reason why we (I) went with kube-lego is that we (I) wanted something that would work, fast, and kube-lego is old enough that it has a pretty decent body of third-party documentation. Cert-manager sounds like it may even be a candidate for going in to kubernetes incubator and so there's a good chance it'll become the de-facto solution. Right now I am really digging that to get green-lock TLS for any project, it's just two extra lines of code in the deployment.yaml.
|
|
#
?
Jan 29, 2018 05:58
|
|
|
Does anyone have experience with Joyent? I've gotten wind that we will be moving from AWS to that in the long term for reasons.
|
|
#
?
Jan 29, 2018 08:59
|
|
|
jaegerx posted:I do openshift if anyone wants help with that.
|
|
#
?
Jan 29, 2018 18:08
|
|
|
Don’t use openshift. Like for real. Don’t use it. It’s not necessary and only complicates running kubernetes. Just use regular kubernetes. VC said kubespray was nice, kubeadm works well enough. Seriously don’t use openshift.
|
|
#
?
Jan 29, 2018 19:11
|
|
|
Yeah the problem with kubernetes is that it does most everything out of the box and creating a vendor specific variant a) locks you in and b) isn't as well suported Unless you're a bunch of data scientist incapable of cobbling together even the most basic k8s cluster using kops (going to check out kube spray) i don't see the point of getting some weird proprietary k8s cluster variant. You get k8s managed for free on GCE, aws will have managed k8s by end of year. Locking yourself in to a vendor specific solution this early in the technology lifecycle seems peculiar unless you're getting kick backs from their sales team.
|
|
#
?
Jan 29, 2018 19:28
|
|
|
I'm genuinely curious what sort of scale folks are running k8s at. Especially those who have been talking about migrating to it or quickly spinning up a new cluster.
|
|
#
?
Jan 29, 2018 22:36
|
|
|
Blinkz0rz posted:I'm genuinely curious what sort of scale folks are running k8s at. Especially those who have been talking about migrating to it or quickly spinning up a new cluster. My day to day is ten nodes each in Dev and prod. The larger org I am part of runs 30 Dev environments with 6-10 nodes and 4 staging/prod with 60-100 nodes each. FWIW I subscribe to the less nodes is better philosophy and my node size is 4cpu 30 gigs at minimum. At my last job I ran 2 environments with 2tb each running at once which varied mode count depending on fleet makeup.
|
|
#
?
Jan 29, 2018 23:04
|
|
|
Blinkz0rz posted:I'm genuinely curious what sort of scale folks are running k8s at. Especially those who have been talking about migrating to it or quickly spinning up a new cluster. https://blog.openai.com/scaling-kubernetes-to-2500-nodes/ Here's another scaling talk from Nordstrom: https://www.youtube.com/watch?v=xZO9nx6GBu0
|
|
#
?
Jan 29, 2018 23:55
|
|
|
I was specifically referring to people posting in this thread but talks about automating k8s and running it at scale are definitely appreciated.
|
|
#
?
Jan 30, 2018 00:18
|
|
|
My poo poo rear end company runs a couple three clusters with ~15 nodes each
|
|
#
?
Jan 30, 2018 02:22
|
|
|
We're running < 5 x t2.medium + 1 x t2.2xl (8cpu + 32gb ram) for some ridiculous stateful data cruncher app that I've massaged in to a stateless-esque service so long as you can deal with the fact that it takes 12 minutes to spin up and ingest 20gb of data. Two clusters like that, one for dev and another for UAT. Prod will likely be slightly beefier but not by much. Probably by the end of the year we might have 10 x t2.medium or 5ish nodes that are roughly double a t2.medium. Running less than 5 nodes gives me the heebie jeebies. Our workflow isn't super dynamic, although once we onboard QA and their ridiculous selenium array it might get more exciting.
|
|
#
?
Jan 30, 2018 03:30
|
|
|
We run ~20 production clusters with between 500gb and 3.5tb of ram, between 5-100 nodes. We're migrating most of our capacity towards hardware, so the cluster sizes will shrink not grow even as our workload increases. We don't have any issues with the 2 or 3 clusters >80 nodes but for a variety of reasons would prefer if they were smaller clusters of higher powered machines. All told we run something like 900 services on them 
|
|
#
?
Jan 30, 2018 03:46
|
|
|
Mao Zedong Thot posted:We run ~20 production clusters with between 500gb and 3.5tb of ram, between 5-100 nodes. We're migrating most of our capacity towards hardware, so the cluster sizes will shrink not grow even as our workload increases. We don't have any issues with the 2 or 3 clusters >80 nodes but for a variety of reasons would prefer if they were smaller clusters of higher powered machines. All told we run something like 900 services on them Wow. How long have you been doing k8s, and what's your cluster management solution? How many people do you have managing one cluster, on average?
|
|
#
?
Jan 30, 2018 06:16
|
|
|
I went from a place that had about 12 m4.2xlarge nodes in production running kops-based k8s to a shop that doesn't even have one application capable of running in Docker, but that's code inertia and letting devs do whatever they want for years though (they'll write super stateful stuff that can't tolerate even a SIGTERM and fail to cleanly shutdown by default). That ran about 3 services - that's right 3 because we had 2 regions, 3 AZs, and each service took about on average 8 GB of RAM (welcome to 8 years of Groovy-powered enterprise shitware). The hardest part of k8s in my experience isn't even deploying it or managing it, it's trying to get clunky old legacy software to be able to take advantage of it via stateless-ish designs. Also, explaining tactfully why you shouldn't put MySQL, Postgres on K8S sans CitusDB or similar to save money. Stateful software isn't impossible but the risks are too much by default. I'm considering using StatefulSets for our current set of services that take 15+ minutes to shutdown oftentimes but finding time is the issue when you're busy rolling clusters one node at a time by hand like it's 1999.
|
|
#
?
Jan 30, 2018 14:22
|
|
|
I guess the thing I'm not looking forward to is working with our platform engineering team to build out a k8s provisioner that actually suits our needs rather than using kops which gets us 80% of the way but leads down a dead end.
|
|
#
?
Jan 30, 2018 14:32
|
|
|
Fwiw I think you can run stateless stuff in k8s. Also if kops has shortcomings in your deploy I’d be really care about that last 20%. You might be fighting hard against the rest of the k8s universe for little gain.
|
|
#
?
Jan 30, 2018 17:53
|
|
|
Punkbob posted:Fwiw I think you can run stateless stuff in k8s. Shortcomings, aka not being able to whitelist security groups rather than cidrs. Kops isn't great, it's just an easy on-ramp. We run a pretty decent production load (anywhere between 1500-2000 instances at any time) and while all of our apps are stateless, moving to an opinionated cluster management system that doesn't lend itself well to CI use and doesn't slot in well with existing infrastructure seems...not good. Deploying services to k8s is one thing but building out and managing clusters at scale is another thing completely.
|
|
#
?
Jan 30, 2018 18:38
|
|
|
Hadlock posted:How long have you been doing k8s, and what's your cluster management solution? How many people do you have managing one cluster, on average? I've been working with k8s for about 2.5 years -- I piloted it at a SaaS company (I left, but they eventually migrated to GKE) and then worked at a your-SaaS-on-prem-with-k8s place, and now I'm at a larger cloud provider. We manage all our clusters and nodes with Chef/Vault/Terraform. No fancy cluster operations thing, but it's really pretty hands off. A team of 6 manages all the clusters, there's not much ongoing maintenance do besides deal with the occasional finnicky dockerd, and upgrade stuff occasionally. Worth mentioning that we have an internal platform for deploying to k8s -- none of our users have direct access to the clusters, they talk to it through a translation layer that takes out some of the complexity of deploying apps and provides out of the box stuff like ingresses, TLS, monitoring, alerting, etc. Pros and cons to that approach, and we're moving away from it -- but it has made onboarding the rest of the organization a lot easier.
|
|
#
?
Jan 30, 2018 19:24
|
|
|
Blinkz0rz posted:Shortcomings, aka not being able to whitelist security groups rather than cidrs. I don’t disagree. K8s touchpoints with existing infra is bad and a lot of my issues that I deal with are related to that. I headcannon it as the k8s devs just expecting to be the only piece of infra to be one way and that’s what they build things towards.
|
|
#
?
Jan 31, 2018 00:39
|
|
|
Redhat bought coreos
|
|
#
?
Jan 31, 2018 02:29
|
|
|
I automated a busted rear end GUI install with AutoIT so we could cut down deployment times by 90%. I have utterly no idea what any of you are talking about, but I’m pretty happy with that.
|
|
#
?
Jan 31, 2018 04:25
|
|
|
Punkbob posted:I don’t disagree. K8s touchpoints with existing infra is bad and a lot of my issues that I deal with are related to that. I headcannon it as the k8s devs just expecting to be the only piece of infra to be one way and that’s what they build things towards. Yeah but that's fundamentally terrible unless you're selling a managed product. Existing infrastructure matters, especially if you're a company that's migrating from a different deployment method. We're likely not going to be able to do a full migration without writing our own k8s chef cookbook and deploying it in the same way we normally deploy clustered services. Sure it'll be nice to cut down on the number of different instance types and decrease the number of instances that run cold but it'll be a ton of work. Not sure how valuable it is at that point beyond speeding up deployments and marginally improving resiliency.
|
|
#
?
Jan 31, 2018 04:34
|
|
|
jaegerx posted:Redhat bought coreos And just announced they're EOL'ing Container Linux. gently caress everything about today.
|
|
#
?
Jan 31, 2018 05:51
|
|
|
jaegerx posted:Redhat bought coreos I’m somewhat pissed* and my whole engineering department isn’t super happy because we really dislike red hat. One of their sales dudes really rubbed us the wrong way and openshift is garbage. *Mainly because the faq already says they want to kill off coreos as a distribution asap and foist atomic on us.
|
|
#
?
Jan 31, 2018 14:13
|
|
|
gently caress everything about this: https://mobile.twitter.com/openshift/status/958454802605846528
|
|
#
?
Feb 1, 2018 02:58
|
|
|
If they manage to keep K8S like they do CentOS v. RHEL it may work out ok. If CoreOS was going to go under anyway for one reason or another then RedHat probably would have taken over it anyway. This way, we get to watch the mass exodus typical of acquisitions and see the resulting distribution (although Illumos has been a bit frustrating to watch given the good stuff in there but so little market traction).
|
|
#
?
Feb 1, 2018 03:16
|
|
|
necrobobsledder posted:If they manage to keep K8S like they do CentOS v. RHEL it may work out ok. If CoreOS was going to go under anyway for one reason or another then RedHat probably would have taken over it anyway. This way, we get to watch the mass exodus typical of acquisitions and see the resulting distribution (although Illumos has been a bit frustrating to watch given the good stuff in there but so little market traction). That’d be a disaster in my view. Right now k8s is independent and openshift is a crappy respin that adds little value, if red hat managed to flip it to a rhel->centos model, I’d be terrified and look to move off of it. It won’t happen though because google still writes a lot of the source code.
|
|
#
?
Feb 1, 2018 03:22
|
|
|
Yeah there is just way too much industry momentum behind k8s for any one company (besides Google) to dominate the conversation, IMO. Whatever Red Hat's social media manager might have to say about things. They'll steer some things but they don't "own" the project anymore than they own the kernel or OpenStack. edit: I agree with you on the random bad taste in my mouth from past Red Hat sales pitches. At my last job the guy that came in to sell us on RHEV was a complete douchenozzle and he scuttled his own deal almost immediately. I don't hate the company top-to-bottom though based on one awful experience. They've done so much for Linux and open source over the years, even if it's lined their pockets at the same time. Never used Openshift, no opinion there. Docjowles fucked around with this message at 04:25 on Feb 1, 2018 |
|
#
?
Feb 1, 2018 04:19
|
|
|
i can't fathom how terrible someone must be at making vendor decisions if they let a douchey sales rep even loving register in the thought process let alone be the key takeaway
|
|
#
?
Feb 1, 2018 05:58
|
|
|
We ended up deploying openstack instead so that wasn't the only questionable decision involved! And idk, if the person whose job it is to describe the merits of their product can't actually do that, and instead spends a couple hours talking about how everyone else's product sucks, it's not exactly getting the vendor relationship off on great footing.
|
|
#
?
Feb 1, 2018 23:10
|
|
|
Is there a good low traffic mailing list for kubernetes
|
|
#
?
Feb 2, 2018 01:34
|
|
|
Hadlock posted:Is there a good low traffic mailing list for kubernetes Hangops slack Or kubernetes slack if you want focused.
|
|
#
?
Feb 2, 2018 02:43
|
|
|
Just use fluentd mannnnnnnnnnnn  Punkbob posted:Hangops slack 
|
|
#
?
Feb 2, 2018 03:42
|
|
|
SeaborneClink posted:Also I know this isn't the first time you've heard this. Goddammnit
|
|
#
?
Feb 2, 2018 06:23
|
|
|
And yet the only goon I've run across in Hangops or the Rands Leadership Slack rarely posts here anymore
|
|
#
?
Feb 2, 2018 17:46
|
|
|
Anyone done anything with CMS automation? We have a bunch of wordpress sites, and I automated infrastructure and a lot of the deployments but of course it sucks. Our main site is Drupal but that was designed by some contracted company and sucks. For some upcoming projects and hires it sounds like we might get more Drupal stuff. I haven't done a lot with it but starting to research and I might try and construct as much as I can using package mangers, composer, hopefully have some tests automatically run. Just hope I have enough time to get some of this sorted out before they start dropping development on people. My org is weird because its not primarily a tech company.
|
|
#
?
Feb 2, 2018 18:01
|
|
|
JHVH-1 posted:Anyone done anything with CMS automation? We have a bunch of wordpress sites, and I automated infrastructure and a lot of the deployments but of course it sucks. The best automation would be giving your credit card to a third-party Wordpress host and letting them deal with it. People are much more receptive to "no, we don't support that" from a third-party host than they are from internal support.
|
|
#
?
Feb 2, 2018 19:47
|
|
|
|
| # ? May 15, 2024 10:33 |
|
|
Vulture Culture posted:And yet the only goon I've run across in Hangops or the Rands Leadership Slack rarely posts here anymore We probably even talked at somepoint, but that's like real life, this is my special space to argue with goons in a well.
|
|
#
?
Feb 2, 2018 20:18
|
|