Blinkz0rz posted:

A 1-2 week delay in wanting to use a dependency to approval?

If so, that's nuts. Putting security as a primary blocker is fundamentally unscalable and only gives engineering orgs ammo to push back against better security measures.

That's why good security teams have increased focus on shifting security left to put that onus on tooling that developers work with directly like npm audit, snyk, etc. and adding gates in the CI/CD pipeline to prevent malicious packages from sneaking through.

The only downside is that it has to be implemented alongside decent endpoint visibility tools and a solid active response process in the event that a malicious packages is identified on a dev's endpoint but if you're putting up a local mirror hopefully you have enough opsec to have those tools and processes in place already.

Bhodi posted:

Yeah, at least it was better than what was previous, which was pushes only on quarterly releases or with emergency approval (two separate approvals up to senior director level signoff). It was a compromise, and security inserting themselves as a gate of any code going into the isolated environment was a mandatory part of the process due to factors outside our control

only having a 1-2 weeks delay from internet was considered a triumph

cum jabbar posted:

Anybody in here self-hosting their Terraform state? We work in a tough regulatory environment and basically need to keep it on our own servers. I'm not sure which storage is most ergonomic and safe.

Hadlock posted:

I am not, we just trust S3 and amazon's promise of the nines

That said, do you have multiple physical locations or just one data center? I'd consider some distributed file system like consul (maybe with vault on top, if you're a real masochist) or ceph

As mentioned, state files are tiny; the default max file size for etcd is 1.5mb but you can increase it to 4mb. Kind of painting your future self into a corner but whatever, 4mb is a lot of state file

Surely someone sells a rack mounted disk appliance that mirrors the s3 api for private datacenter use? Just use that? Even google and azure's s3 products use the s3 api.

Blinkz0rz posted:

You could run localstack on a server in your DC and make sure the disk is mirrored.

cum jabbar posted:

In this environment, it would be a problem if teams had the ability to access each others' secrets, either by accident or malicious intent. Got to be sure about the default security settings on our S3 offering.

cum jabbar posted:

Anybody in here self-hosting their Terraform state? We work in a tough regulatory environment and basically need to keep it on our own servers. I'm not sure which storage is most ergonomic and safe.

Methanar posted:

Your security posture allows you to use terraform to create presumably public cloud resources, but disallows you to store state in s3?
What kind of regulatory environment is, this exactly.

But whatever, if this is a regular thing for you, I'm a bit surprised you don't have your own self hosted swift/minio/s3 compatible object storage already which would be the next best thing.

cum jabbar posted:

Anybody in here self-hosting their Terraform state? We work in a tough regulatory environment and basically need to keep it on our own servers. I'm not sure which storage is most ergonomic and safe.

cum jabbar posted:

In this environment, it would be a problem if teams had the ability to access each others' secrets, either by accident or malicious intent. Got to be sure about the default security settings on our S3 offering.

Hadlock posted:

I mean, if you start with k8s and only design 12 factor apps, it's pretty straight forward

Backing 11 years worth of badly written code and questionable architecture decisions that "mostly work as long as everything is on the same server" into 12 factor and then containerizing, then deploying k8s via terraform is painful, sure

Once you learn the 12 factor/container/k8s pattern and train up engineering to deploy new services in a sane and consistent manner, managed k8s is like greased lightning. Sorry you feel otherwise

Vulture Culture posted:

, what have you bought them? Throw that poo poo in App Runner or Fargate or something.

duck monster posted:

So recently I started a new job that partially involves inhereting a giant Kubernetes cluster on DigitalOcean. I've never used Kubernetes so its all a massive learning curve.

This moring I got into the office and realised the entire cluster down with all the pods in "Pending" mode (Including about a bazillion cronjob containers that seemed to be piling up).

It would seem at some point in the night for reasons I'm completely unsure of the whole drat thing was reset causing it to reissue a whole bunch of nodes which where in an unlabeled state.
So after labelling them, it all came back up, although I had to delete the node spec for the cronjobs because there where literally hundreds of the bloody things trying to be created. Followed by a slow recycling of nodes to get the drat things to exit the "Terminating" state.

Massive and disrupive pain in the arse.

Is there a way to tell Kubernetes how to label nodes after a rebuild? Beause this *sucks*

my homie dhall posted:

The process you should be looking at is kubelet. Looks like you can modify the kubelet config to have the kubelet come up with whatever node labels you want.

necrobobsledder posted:

According to this Github issue there's a CLI tool you can use to set the node labels https://github.com/digitalocean/DOKS/issues/3

Newf posted:

someCommand is running on MY computer.

How do I execute someCommand on someComputer from inside a script? (A GH action again...)

Hadlock posted:

Yeah I played this game at my last job

"Why use kubetnetes? It's so complex! Just use aws batch for batch jobs!" Ok so now Ben is the AWS batch job expert, it has it's own idiosyncrasies, so if you have any trouble with it, go bug him since nobody else knows it.

"Screw kubetnetes, we're running our container on fargate. Out team owns our infrastructure and this is the non standard tool we use, and it's going to take three weeks to onboard you to our workflows, so watch out if you want to join our team"

"Hey we're the back end team, we aren't bandwagon people so we went with ECS. Talk to Fred over there, he sort of knows Ben and they got our backend working over on ECS so as long as neither of them get hit by a bus the company is safe. It uses a different configuration methodology from AWS batch and fargate though, so we only allow transfers to this team if you already know ECS. Gosh why is it so hard to find people with ECS experience?"

"Why can't we hire and why is our velocity so low? What do you mean each team uses an entirely different tool with completely different configuration philosophies, what do you mean you can't deploy our entire stack with a single command line? Oh man we are super vendor locked in, there's not a single engineer who understands how our entire stack is deployed across the cloud"



After I left they merged with another company and their CTO gave them six months to transition everything to kubetnetes so that all the engineers could work across the entire stack and services could be shared

Kubetnetes is not perfect but it gives you a cloud agnostic lingua franca to deploy code in a consistent manner and works for an overwhelming majority of companies

Hadlock posted:

Edit: current company, 95% of our deployments are in kubetnetes and we were ready to do a major DR project but got halted when we realized this other highly siloed team is running their poo poo on elastic beanstalk and it's going to take a couple of quarters to get them up to snuff with the rest of the company so they're actively holding back that project and they're not compatible with the rest of the company's deployment/IAC yay

Vulture Culture posted:

You're arguing from the standpoint of two things, which are different, being the same thing. I would like to argue the opposite: ...and the problem you're describing.

Hadlock posted:

You might want to look into buildah/podman to streamline your build process, instead of docker in docker

x1o posted:

Kaniko is also another solution you might want to look at, my area uses it as our replacement for docker in docker

Methanar posted:

https://github.com/kedacore/keda/blob/d5b29339c46b7635a568ff65013ea212604bde9f/pkg/scalers/prometheus_scaler.go#L211
https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podautoscaler/horizontal.go#L375-L395

https://pkg.go.dev/k8s.io/api/autoscaling/v2beta2#MetricTarget

code:

	// averageValue is the target value of the average of the
	// metric across all relevant pods (as a quantity)
	// +optional
	AverageValue *resource.Quantity `json:"averageValue,omitempty" protobuf:"bytes,3,opt,name=averageValue"`

lol keda. Hardcoded to treat the response of any Prometheus query as a per-pod average even though that's loving stupid.

This has been a nightmare to track down why applications are failing to scale up in response to kafka lag sometimes.
It's because at low pod counts, that per-pod average is dividing by a small number. At large pod counts, it divides by a large number.
If I had a threshold of 50k unprocessed messages. Well if I'm at a milliion lag dividing that by 10 pods is still 100k lag when the threshold is 50k. nbd it will work.
I have a lag threshold of 50k and a million lag divided by 100 pods, well that's an average of 10k lag per pod. less than the threshold of 50k and nothing will happen.
of course, the act of scaling up increases the number of pods. So you will see the app scale up at first, but eventually stop scaling up due to the above scenario. for some reason this is never at all once in the documentation identifying that despite being Kind Of Important.


The kafka scaler does the same thing lmfao
https://github.com/kedacore/keda/blob/d5b29339c46b7635a568ff65013ea212604bde9f/pkg/scalers/kafka_scaler.go#L359
Who cares about lag per pod. I care that there is lag.

Vulture Culture posted:

What I think you actually want is to SLA the throughput of each of your consumers, and if one of your consumers falls below the expected rate—i.e. you have single identifiable partitions that are underperforming the others—you kill the consumer and replace it with one that works correctly. You then couple that with pod autoscaling off the average.

Vulture Culture posted:

My gut here is that what you're trying to do is completely the wrong approach for Kafka, and this is hardcoded this way to protect you from a bad idea.

If you get an irregular latency distribution because you have, say, a single slow consumer that's failing to drain a single partition, there's no reasonable expectation that increasing your number of consumers is going to even help the lag.
Your topic has a finite number of partitions, and your consumers consistently hash onto them. One partition has at most one consumer. What I think you actually want is to SLA the throughput of each of your consumers, and if one of your consumers falls below the expected rate—i.e. you have single identifiable partitions that are underperforming the others—you kill the consumer and replace it with one that works correctly. You then couple that with pod autoscaling off the average.

quote:

What I think you actually want is to SLA the throughput of each of your consumers, and if one of your consumers falls below the expected rate—i.e. you have single identifiable partitions that are underperforming the others—you kill the consumer and replace it with one that works correctly. You then couple that with pod autoscaling off the average.

Methanar posted: