|
CMYK BLYAT! posted:rework a bit of systemd so it will run within containers and manage their local processes. people (terrible, horrible people) have been doing this for several years, since approximately 5 seconds after it became possible. of course they have freeasinbeer posted:now the termination notification of the pod also removes it from the service IP, so you should have 30 seconds of no traffic before the sigkill this is not always true and it can take a while for the NotReady to propagate, at least for NodePort services
|
#
¿
Dec 26, 2019 17:17
|
|
|
|
| # ¿ May 8, 2024 04:59 |
|
|
Jonny 290 posted:a genuine question for all the k8s wranglers: fleetctl? idk maybe nothing by the time I started mangling containers for money k8s had already won so I’ve never touched fleet what’s wrong with docker? that whole thing where it’s a pile of poo poo. that’s pretty bad. cri-o will save us someday
|
|
#
¿
Dec 27, 2019 07:25
|
|
|
if you’re on AWS you qualify as a network engineer for knowing what a default route is
|
|
#
¿
Dec 28, 2019 00:55
|
|
|
MFA with radius is a shitshow
|
|
#
¿
Dec 29, 2019 05:17
|
|
|
abigserve posted:(some VPN' solutions use Radius to auth the user, then LDAP to then map that user to groups, for example). fuuuuuck palo alto e: palo alto also uses ldap for group mapping when your auth method supports returning group membership, e.g. saml. every single thing about the product is like that
|
|
#
¿
Dec 30, 2019 01:13
|
|
|
abigserve posted:nah they'll map groups onto fuckin anything if there is a username in the db and group mapping is configured. You can do it that way too though, where the authentication server returns the group membership but then it depends on the protocol (RADIUS/SAML/whatever) like hell am I gonna dig through the docs again but this is something that was specifically called out. I remember because I was still at a point where I was capable of being astonished by Palo Alto. you can do saml auth for globalprotect but the firewall will ignore any groups procided by the saml callback. also groups obtained from group mapping can’t be used as gateway selection criteria, presumably because group mapping hasn’t happened yet
|
|
#
¿
Dec 30, 2019 02:15
|
|
|
Turnquiet posted:my radius beef is when apps try to use if to user authn. the stuff i build with my stupid fartastic cloud deployments is premised upon ephemeral infrastructure, so when loving cyberark wants to use radius for user auth and my pam guy just assumes i will turn on my pingfederate's radius capabilities i get irked because i am stupidly trying to get us into the cloud like what the last 8 years of c-levels have said is the strategy, which is a place where we can't guarantee a fixed ip address. gently caress you, in preparation for zero trust we are using federated protocols for all authentication, so use saml2 or oidc or die in a fire- looking at you, microsoft, who literally built azuread on openid flows but still demands ws-trust/ws-fed if you want to retain control of your idp. are you on aws, because you definitely can get a fixed IP on aws
|
|
#
¿
Dec 30, 2019 04:52
|
|
|
graph posted:but this costs money nah just dont delete the eni
|
|
#
¿
Dec 30, 2019 05:03
|
|
|
Turnquiet posted:you get 5 elastic IPs per account, and they cost money and are usually (and justifiably) reserved by your cloud folk for some other, stupider use cases. just counting on never destroying the eni almost sounds like it would work if you didn't design your cluster for scaling and you connected directly to a combo admin/engine instance. for a clustered deployment you have n engines fielding requests, and those nodes attach to an nlb which is what the dns resolves to. and you don't get to control when/how was cycles your ip addresses behind your nlb/albs. how much radius are you doing that a single instance isn't enough, holy poo poo
|
|
#
¿
Dec 30, 2019 18:13
|
|
|
Cocoa Crispies posted:Comcast has a nationwide wpa2 enterprise network they need to do more than several thousand requests/sec?
|
|
#
¿
Dec 30, 2019 18:58
|
|
|
Cocoa Crispies posted:multiple instances aren't just for throughput and radius isn't just for yes/no authentication, so if there's a bunch of like accounting traffic that needs to be sharted that's going to mean you need to fan out those requests to a handful of hosts so it gets a load balancer and a dozen instances because thats how things are done in 2019. ok, fair enough
|
|
#
¿
Dec 31, 2019 01:25
|
|
|
Ploft-shell crab posted:any of y’all tenants trying to get you to run a god dang “service mesh”? idk what real problems these things are trying to solve, I think they’re just inventing stuff for themselves to do load balancing and canaries. basically, threading a single end user request back through a load balancer 10 times because of microservices is obviously horrible. service mesh isn’t obviously horrible
|
|
#
¿
Feb 12, 2020 20:02
|
|
|
the talent deficit posted:service meshes are for when you've given up on your developers giving a gently caress about monitoring, reliability or observability its something the platform team can roll out to improve poo poo across the board without messing with product teams. so like yes you're right but also this is how we want it i guess
|
|
#
¿
Mar 2, 2020 05:52
|
|
|
the talent deficit posted:i mean i get it but it's like installing inflatable bumpers along roadsides because drivers keep driving off the road. it's a terrible solution to a terrible problem that has a much simpler solution (ban cars/microservices) as a result of breaking a bunch of services out from a big ol monolith, our half-dozen or so product teams can deploy independently, and more importantly can rollback independently. rolling out linkerd so we can do canaries between the services is treating a self-inflicted wound, but pulling everything back into a single process would be even worse. if you have a Third Way architecture the resolves all these issues, please do share it and I will be happy to present it as my own at work and collect the kudos for solving a pretty significant problem we're facing
|
|
#
¿
Mar 3, 2020 00:16
|
|
|
suffix posted:google charging per gke cluster now https://cloud.google.com/kubernetes-engine/pricing lol what are you doing that this matters. $75/cluster/mo is basically nothing in any sane k8s deployment scenario
|
|
#
¿
Mar 4, 2020 23:13
|
|
|
psiox posted:while the per-hour cost isn't that crazy, i thought that anybody using kubernetes is actually running N^2 kubernetes cluster instances to test that their poo poo won't break we’re running 6 clusters, but the EKS fees on 36 clusters would still be less than 5% of our overall spend. still at the “this is not what bankrupts us” level CMYK BLYAT! posted:we use it primarily to test and develop k8s tooling in a realistic environment, so it's just 3 workers and the extra cost is significant for that I’m not sure how you get from “large number of tiny clusters” to “our testing is happening in a realistic environment”. I assume you have a large number of clusters, anyway, because otherwise why gaf
|
|
#
¿
Mar 5, 2020 05:50
|
|
|
sure, it’s great until google loses interest and shuts it down. that’s scheduled for 2023 iirc
|
|
#
¿
Apr 20, 2020 14:14
|
|
|
klosterdev posted:Literally made one of these with Ethernet cable and phone heads sudo su: 2 strokes unannounced reboot: 5 strokes running a wow bot farm on the ML cluster: 25 strokes
|
|
#
¿
May 25, 2020 04:12
|
|
|
animist posted:so i dont actually know what service meshes are so i googled it and... is it just people reimplementing packet switching in software over a bunch of VPNs?? whyy does anybody need this you nailed it. nobody ever needs it. service meshes are a communist plot to degrade our precious adtech microservices’ latency and throughput
|
|
#
¿
May 27, 2020 05:28
|
|
|
carry on then posted:you've never worked on anything at scale have you? why are you using puppet, it’s a waste of time writing all those manifests. just back up the server regularly
|
|
#
¿
May 27, 2020 05:34
|
|
|
if k8s on aws provided a LoadBalancer service with support for http2 we’d probably not bother with linkerd
|
|
#
¿
May 27, 2020 18:11
|
|
|
I rolled out k8s more or less singlehandedly at my last job and it was fine. poo poo, I wasn't even doing infra full time. It's not something that a fullstack dev who just installed Ubuntu for the first time last Thursday is going to be able to do successfully, though. It also requires being willing to adapt your requirements to what is easy and effective, instead of going full Aspie and insisting that everything must be exactly how you want it.
|
|
#
¿
Jun 9, 2020 06:49
|
|
|
k8s doesn’t give you anything that sufficiently good cloud tooling can’t do directly. in-house tools are usually poo poo though, which makes k8s valuable almost everywhere. the downside is k8s sucks dog dicks and this will cause lots of pain if you have needs that are in any way unusual. such as, it’s not actually possible to guarantee that a pod stops receiving requests before it stops. something that any sysadmin with a browser open to the EC2 api reference and a working copy of bash can script in 15 minutes, if you’re running on VMs directly
|
|
#
¿
Jun 10, 2020 04:19
|
|
|
uncurable mlady posted:well that's one of the tradeoffs, innit? i don't think it's unreasonable to suggest that if you're building software to run on k8s, that your devs should be immune from having to learn how it works and add in a wait or something on SIGTERM to keep in-flight requests from failing. that would be the suggestion of kubernetes devs, yes, that you add in an unconditional wait before sigterm is sent. sometimes this is fine. it makes everything slower, which might be why it doesn’t happen by default adding a sleep is still not the same thing as guaranteeing that traffic has stopped, which again is trivially achievable if you’re using ELB and EC2 yourself. it’s the layer of iptables redirection poo poo in between required by k8s’s networking model that causes the problem. that, and the k8s devs insisting that it’s a layering violation if the left hand knows what the right hand is doing. SIGKILL a pod even though requests are still getting sent to it by several cluster nodes? sure! we don’t have a large prod cluster, maybe 100 nodes, but some node not updating its iptables rules and still sending traffic to a dying pod for 1+ _minutes_ after the pod started termination is a regular occurrence. this is entirely an unnecessary consequence of k8s’s architecture and nothing else, and it makes it supremely aggravating to run services that need to be reliable on top of k8s
|
|
#
¿
Jun 10, 2020 18:53
|
|
|
12 rats tied together posted:IMO its fair that, as a google product, k8s has high expectations of you in terms of your overall ability to control the applications that you manage. for something like a ui node, dropping inflight requests shouldn't matter because your web ui is just a javascript api client bundled with static content, so you can cleanly handle retry logic behind the scenes. we do realtime money things for large companies that care very much about SLAs. neither retries nor just not giving a gently caress about errors are options for us, which is a huge problem when using k8s (motto: if it ain’t broke, it needs more yaml)
|
|
#
¿
Jun 10, 2020 19:01
|
|
|
we’re closer to adtech than stocks. when k8s was rolled out here, everything ran in batch and retries were fine restart pods whenever you want, sure. nobody cares if something falls on the floor temporarily. but then some very large deals were only possible if we could do stuff realtime instead, CTO decided we would go for it, and the pain started
|
|
#
¿
Jun 10, 2020 20:03
|
|
|
it’s fraud prevention. the performance requirements and system design are the main similarity to adtech
|
|
#
¿
Jun 10, 2020 20:16
|
|
|
TwoDice posted:if your system requires that servers be politely shut down to work properly then it's a bad system pray tell me, master, how shall I rebuild this system such that requests succeed even when the server has been SIGKILLed with the request in flight
|
|
#
¿
Jun 10, 2020 21:08
|
|
|
Progressive JPEG posted:could put the requests on a queue, and commit the queue as requests are successfully processed. but i think this would usually give "at least once" guarantees when it sounds like you want "exactly once"? when I said realtime earlier I meant we have a 250ms deadline. request comes in from merchant, we do as much fraud detection as we can in that window, return a decision. if we hit 230-240ms without reaching a conclusion then we return a random decision. that leaves very little room for timeouts and retries. obviously there are ways to decouple processing from the success or failure of a single http request, but they’re not applicable here. we get sent one request and it’s our one chance to do something reasonable. getting the client to retry isn’t possible because waiting for us is blocking them taking payment and showing a spinner in the customer’s browser these are very harsh constraints! sometimes poo poo doesn’t work right and we fall back to random or even time out. poo poo happens and you just have to accept that. it’s just aggravating as gently caress to discover that k8s is built for workloads that no one really gives a poo poo about except at large scales. we do actually want to avoid chopping off those dozen requests if we can, because high-value merchants are watching that success rate very closely, and this flies directly in the face of everything k8s stands for
|
|
#
¿
Jun 10, 2020 22:00
|
|
|
Captain Foo posted:if you are a successfully serving internet ads you are unironically contributing to the downfall of society rcp
|
|
#
¿
Jun 11, 2020 01:04
|
|
|
abigserve posted:Seems like you need middleware (or better? middleware) between the client and the backend to buffer the request and determine if the backend is hosed yeah, you’re not saying anything I don’t know. it’s not feasible to guarantee that requests never fail. I’m bitching about an aspect of k8s that guarantees some requests will fail, when they could have made different choices to avoid the failures. architectural purity is valued more highly than proper operation, and that pisses me off
|
|
#
¿
Jun 11, 2020 01:07
|
|
|
CMYK BLYAT! posted:"possible" is a strong word. there are some things that are indeed not possible unless you throw a whole lot of things out the window--if you want to exceed c and violate causality, for instance, some very fundamental things need to change, so that's de facto not possible. computers are usually a bit more flexible. possible is the right word. clusterip services are enabled by local state on each worker node, and it's not possible to determine from the apiserver's perspective whether a node is up to date, so it's not possible for a kubelet to know whether it's safe to kill a pod. it is literally impossible to get the semantics of elb:DeregisterInstancesFromLoadBalancer out of k8s, and adding sleeps doesn't fix the race. solving this problem was too hard for the brain geniuses making k8s to do, so they didn't, and you're not supposed to care, because you're supposed to be hosting garbage that no one will notice if it breaks occasionally, like google's thirtieth chat app but thank you for talking down to me and linking me to a youtube that recapitulates the documentation at me while breathing into the mic. being told by some guy at a con that i need to do what i already did six months ago has fixed all of my problems
|
|
#
¿
Jun 11, 2020 03:02
|
|
|
uncurable mlady posted:maybe your requirements are bad? idk, i don't do ecommerce or fraud detection so i trust that y'all have thought about this more than i have. can you understand me being pissed that the best practice guidance for this situation is to hack in a sleep long enough that you probably won’t have an issue, because making the thing work properly would be a “layering violation”. it’s worse is better pretending to be just plain better and it bugs the hell out of me
|
|
#
¿
Jun 11, 2020 06:24
|
|
|
ate poo poo on live tv posted:I dont' understand, you can't just put your k8s cluster behind a load balancer, then remove the cluster or node from the LB pool which will then no longer send requests to the cluster, then 30sec latter send a sigterm? in what universe is it sane to spin up a second cluster and write my own LB glue so that I can safely restart a single process. and that doesn’t address intracluster traffic at all
|
|
#
¿
Jun 11, 2020 08:14
|
|
|
12 rats tied together posted:its a fair gripe, this app probably shouldnt be in k8s prob not but at the time k8s was implemented we had a 12 hour sla and cared extremely little about failure rates. retries will fix everything. now we have some features that care very much about latency and failure rate but are all-in on k8s so now making do is part of my job Ploft-shell crab posted:the authors of kubernetes would probably say you’re going to have to solve this problem (requests dropping) eventually somewhere and also the problem you want them to solve is intractable. you’re asking them to somehow come up with a routing model that’s a) consistent/atomic across distributed nodes, b) supports dynamic scaling events and c) never dropping requests. how would you do this with any other provider or even conceptually other than a sleep after scaling down? your a) is stronger than I need. what I’m looking for is an ordering guarantee between iptables updates and SIGKILL. you don’t need consensus for that. write a txid into endpoints when they’re updated, and write the most recent txid visible in iptables into node status. the rest is a pretty simple pre-stop hook. paging someone to unfuck the cluster when a node breaks is acceptable to me. god knows we do that enough with loving docker as it is
|
|
#
¿
Jun 11, 2020 16:46
|
|
|
obtw 12 rats can you point me to where on the k8s marketing site or docs it says that kubernetes is unsuitable for applications requiring correctness from the cluster orchestrator. I mean, we’ve just established that it is, so isn’t that something that is pretty important to communicate to people considering using k8s
|
|
#
¿
Jun 11, 2020 16:49
|
|
|
uncurable mlady posted:i don't think kubernetes has ever claimed that it supports absolute correctness in the way you want it to. how often do you publicly claim not to beat your spouse and should i derive any conclusions from the frequency of such claims
|
|
#
¿
Jun 11, 2020 17:39
|
|
|
kube saying "im so wasted lol is it time to murder this pod, idk lets just do it yolo" with no way to change it, i just wanna bitch, and yall bein like 1) here is a thing you are already doing, have you tried it 2) actually this is good, you should stop complaining 3) i would simply do a 360 and rebase my entire infrastructure to a different system what you should say is "drat bro, that sucks"
|
|
#
¿
Jun 11, 2020 17:50
|
|
|
uncurable mlady posted:idk i feel like a lot of people keep saying 'drat bro, that sucks, but here's some productive options and discussion about the issue' but if you just want to scream into the void might i recommend using the no-reply tweets to @thockin you're the one who just got done saying i should either shut up and suck it up or spend a few man years to move off of k8s, so it's really interesting to me that this is how you see your side of it
|
|
#
¿
Jun 11, 2020 18:13
|
|
|
|
| # ¿ May 8, 2024 04:59 |
|
|
Ploft-shell crab posted:just because a host no longer has an iptables rule for a target doesn’t mean it’s not going to route traffic to that target though. existing conntrack flows will continue to use that route which means you’re not just waiting on iptables to update, but also all existing connections (which may or may not even be valid) to that pod to terminate before you can delete it. what happens if neither side closes the connection, should the pod just never delete? sure, no L4 load balancer can help you much there. it doesn't know how to gracefully shut down a connection, but that's fine. kube gives you the tools out of the box to tell a process it's about to die, so it can drain itself and shut down gracefully assuming will never be sent a new connection. what i want and don't have is a guarantee that a pod won't die until after every node has stopped sending it syn packets. you can't do a graceful shutdown without that, not safely. i'm willing to accept pods living longer than they need to or occasionally hanging in terminating with manual intervention required to fix, so yeah I think the process i outlined would work without adding too much write load on the apiserver.
|
|
#
¿
Jun 18, 2020 00:33
|
|