Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender

Gyshall posted:

Anyone have recommendations for an 2FA hardware token that I can poll with an API call?
Yubikeys? Pretty sure you can configure them in a way that you don't have to push the button to activate them. We had a release signing server that had a Yubikey plugged into a motherboard USB port.

Adbot
ADBOT LOVES YOU

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.
Gotcha, I'll look at that. I was hoping something cloud based... Ideally my script runs, gets to a point where it will check for a 2fa approval or two, and then continue running.

minato
Jun 7, 2004

cutty cain't hang, say 7-up.
Taco Defender
Perhaps AWS HSMs then? https://aws.amazon.com/cloudhsm/

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.
Yeah that is neat - thanks.

Anyone here using Netbox with SAML/Okta? We're trying to get it working and it's been a gd nightmare for us. Was hoping someone had a working reference implementation.

drunk mutt
Jul 5, 2011

I just think they're neat

Gyshall posted:

Yeah that is neat - thanks.

Anyone here using Netbox with SAML/Okta? We're trying to get it working and it's been a gd nightmare for us. Was hoping someone had a working reference implementation.

Seeing as Netbox is just a Django application, you should be able to find a crap load of documentation on Django+Okta and I know for sure there are some modules for SAML that (at least use to) integrate with Django's auth framework.

namlosh
Feb 11, 2014

I name this haircut "The Sad Rhino".
So, is this the thread to ask about container stuff? I looked but couldn't find one...

So, what runtime should I be using to mess around with? should I not be using docker cli to run containers? I'm thinking just home lab environment for now.

I'd like to use containerd and maybe cri-o runtime, is there a cli recommended to run with that?

Like, with the death of docker (NOT hub.docker, which is very much alive), when I'm messing around with images and such, should I still be typing:

$: docker run...blahblahblah

or is there something better?

my homie dhall
Dec 9, 2010

honey, oh please, it's just a machine
docker is totally fine to use and likely you won’t notice any difference between runtimes except that docker will be easier to find resources online for

my homie dhall
Dec 9, 2010

honey, oh please, it's just a machine
container runtime is like the least interesting thing to spend time thinking about unless you are executing arbitrary untrusted code

namlosh
Feb 11, 2014

I name this haircut "The Sad Rhino".

my homie dhall posted:

container runtime is like the least interesting thing to spend time thinking about unless you are executing arbitrary untrusted code

point taken, thanks!

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.
Anyone know of a service where I can have sms sent to a number and read the content over api?

tortilla_chip
Jun 13, 2007

k-partite
twilio

whats for dinner
Sep 25, 2006

IT TURN OUT METAL FOR DINNER!

Gyshall posted:

Anyone know of a service where I can have sms sent to a number and read the content over api?

Like tortilla_chip said: Twilio. They would be my first port of call for pretty much any programmatic SMS and voice. If they're no good for some reason then both Vonage and Sinch have decent APIs.

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.
We use twillio now for our app which works good for outgoing... The team wanted to find something non twillio for our qa team to use for programmatic validation of text messages. I already suggested the twillio api which looks like it can do this so I'll check out the other suggestions too. Thanks goons

Methanar
Sep 26, 2013

by the sex ghost
Anyone else eat poo poo today because quay went down?

freeasinbeer
Mar 26, 2015

by Fluffdaddy

Methanar posted:

Anyone else eat poo poo today because quay went down?

:waves:


We also accelerated rollout of an admission controller that checks if an image is in ECR, and then mutates the image to match the ecr location of it is. If not it sends to sqs and we use skopeo to sync it. Doesn’t help if the image isn’t already cached but if it is it’s dope.

Methanar
Sep 26, 2013

by the sex ghost

freeasinbeer posted:

:waves:


We also accelerated rollout of an admission controller that checks if an image is in ECR, and then mutates the image to match the ecr location of it is. If not it sends to sqs and we use skopeo to sync it. Doesn’t help if the image isn’t already cached but if it is it’s dope.

We have an company-wide adoption project to adopt artifactory and we had a line item as part of that to use artifactory as a pull-through cache for all docker images. Too bad it was still 3-4 weeks out :haw:

skopeo looks interesting. Never heard of this before.

It's all a bit frustrating because recently as part of the docker hub rate limiting thing I had to move a bunch of stuff over to quay. And now I moved a bunch of stuff off of quay...

One of the things recently was I added an init container to our kiam stuff that was originally busybox. But that obviously failed soon after because upstream dockerhub rate limited me. So I then changed it to quay. And now I had to change it again to something hosted internally that I basically manually copied over.

I spent like 6 hours dealing with this today.

Methanar fucked around with this message at 03:23 on Mar 23, 2021

Methanar
Sep 26, 2013

by the sex ghost
I'm still getting pages coming in because one of the other things I had to replace was node-exporter. and that meant rolling prometheus operator chart everywhere.

And part of my changes of that included kube-state-metrics. But because we suck somebody on my team told me not to do kube-state-metrics because he looked upstream and it was using gcr. But only on the master branch and not the tag release we were actually using. So it turns out I did need to update kube-state-metrics after all. Except I only realized that like, 30 minutes ago and I'm not dealing with that dance again just for kube-state-metrics at 8pm when my existing kube-state-metrics singleton is fine, blocked on being terminated, because the replacement can't pull its non-updated image from quay.

So now I'm just acking the pages for 24h until it fixes itself.

kube-state-metrics

Also it was a giant pain in the rear end because this morning we staged kiam 4.0 in a test environment and some other prom operator changes and I ended up needing to do all these different things to account for that after I realized that we had partial state in different environments, but the PRs were all merged to master as we waited for the changes to be used in a dev environment for a day before promoting to prod. kiam 4 was a major pain in the rear end specifically because upstream made a breaking change to replace whitelist with allow in an argument to be PC and that burned a good 30 minutes of my time screwing around to realize what happened and to fix all my git branches to actually represent everything properly such that I could maintain 4.0 in dev, but stay on 3.6 in prod.

Methanar fucked around with this message at 03:33 on Mar 23, 2021

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Methanar posted:

I'm still getting pages coming in because one of the other things I had to replace was node-exporter. and that meant rolling prometheus operator chart everywhere.

And part of my changes of that included kube-state-metrics. But because we suck somebody on my team told me not to do kube-state-metrics because he looked upstream and it was using gcr. But only on the master branch and not the tag release we were actually using. So it turns out I did need to update kube-state-metrics after all. Except I only realized that like, 30 minutes ago and I'm not dealing with that dance again just for kube-state-metrics at 8pm when my existing kube-state-metrics singleton is fine, blocked on being terminated, because the replacement can't pull its non-updated image from quay.

So now I'm just acking the pages for 24h until it fixes itself.

kube-state-metrics

Also it was a giant pain in the rear end because this morning we staged kiam 4.0 in a test environment and some other prom operator changes and I ended up needing to do all these different things to account for that after I realized that we had partial state in different environments, but the PRs were all merged to master as we waited for the changes to be used in a dev environment for a day before promoting to prod. kiam 4 was a major pain in the rear end specifically because upstream made a breaking change to replace whitelist with allow in an argument to be PC and that burned a good 30 minutes of my time screwing around to realize what happened and to fix all my git branches to actually represent everything properly such that I could maintain 4.0 in dev, but stay on 3.6 in prod.
What are the advantages for y'all in kiam vs. IRSA?

Methanar
Sep 26, 2013

by the sex ghost

Vulture Culture posted:

What are the advantages for y'all in kiam vs. IRSA?

Kiam came out first.

Kiam is a gross fragile single point of failure hack that works by MITMing traffic with iptables dnat rules.

We're getting rid of it in favor of IRSA soon with this https://github.com/aws/amazon-eks-pod-identity-webhook/

my homie dhall
Dec 9, 2010

honey, oh please, it's just a machine

Methanar posted:

a gross fragile single point of failure hack that works by MITMing traffic with iptables dnat rules.

so, uh, just like kube-proxy?

namlosh
Feb 11, 2014

I name this haircut "The Sad Rhino".
I have a lot of googling to do to understand the last couple of posts


kube-state-metrics

Methanar
Sep 26, 2013

by the sex ghost

my homie dhall posted:

so, uh, just like kube-proxy?

kube-proxy isn't quite the same as its just a control plane component. If kube-proxy starts crashing, traffic still flows properly since the real datapath is through iptables which persist.

If kiam goes down, all requests for API tokens fail and you have an immediate problem.

freeasinbeer
Mar 26, 2015

by Fluffdaddy
Fwiw, across 3 companies, artifactory has just poo poo the bed with docker images, blobs will just start corrupting.


I am not a huge fan of it.

my homie dhall
Dec 9, 2010

honey, oh please, it's just a machine

Methanar posted:

kube-proxy isn't quite the same as its just a control plane component. If kube-proxy starts crashing, traffic still flows properly since the real datapath is through iptables which persist.

If kiam goes down, all requests for API tokens fail and you have an immediate problem.

ah, true

The NPC
Nov 21, 2010


I'm in a heavily Windows shop and k8s has veen declared ~the way of the future~. We have done a pretty good job of moving devs off of TFS/TFVC and on to Azure DevOps/git. CI/CD has rocked their world and made us (Ops) look like wizards.

Something I'm having trouble wrapping my head around though: a lot of our apps are various versions of aspnet webforms/mvc hosted on IIS. We have been able to move most of the apps off of integrated windows auth and on to either okta or azure ad. How are people handling integrations dependent on the app pool identity? E.g.: site1 runs under a service account and accesses smb file shares as that identity.

I see k8s has support for gmsas now. Has anyone had success with them? For the apps that have been ported to dotnet core, is there a way to do this integration from linux containers?

Methanar
Sep 26, 2013

by the sex ghost

The NPC posted:

I'm in a heavily Windows shop and k8s has veen declared ~the way of the future~.

What is the business case behind this?

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.

The NPC posted:

I'm in a heavily Windows shop and k8s has veen declared ~the way of the future~. We have done a pretty good job of moving devs off of TFS/TFVC and on to Azure DevOps/git. CI/CD has rocked their world and made us (Ops) look like wizards.

Something I'm having trouble wrapping my head around though: a lot of our apps are various versions of aspnet webforms/mvc hosted on IIS. We have been able to move most of the apps off of integrated windows auth and on to either okta or azure ad. How are people handling integrations dependent on the app pool identity? E.g.: site1 runs under a service account and accesses smb file shares as that identity.

I see k8s has support for gmsas now. Has anyone had success with them? For the apps that have been ported to dotnet core, is there a way to do this integration from linux containers?

Good luck.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

Gyshall posted:

Good luck.

This. Windows containers are trash and I bet they suck in kubernetes.

Hadlock
Nov 9, 2004

New Yorp New Yorp posted:

This. Windows containers are trash and I bet they suck in kubernetes.

Spring Heeled Jack
Feb 25, 2007

If you can read this you can read
We run a bunch of .NET core microservices in AKS (Linux nodes) with no issue. All of our full framework poo poo is in IIS and probably will be forever because what is technical debt?

The lead dev brings up trying Windows nodes in AKS occasionally and I always change the subject quickly.

12 rats tied together
Sep 7, 2006

It's pretty easy to mount like, a cifs share or something, from a linux container. There might even be a native volume mapping type for that in the k8s spec already.

This is probably entirely out of your control but all the windows identity stuff is extremely crappy and dated. The sooner you can solve this problem by "not having apps access anything with app pool identity", the better.

e: If you're lucky you can just delete your AD domain too, while you're at it

The NPC
Nov 21, 2010


Methanar posted:

What is the business case behind this?
:shrug: We have a new VP who wants to build in-house PaaS offerings for all of our Ops services.

12 rats tied together posted:

This is probably entirely out of your control but all the windows identity stuff is extremely crappy and dated. The sooner you can solve this problem by "not having apps access anything with app pool identity", the better.
Care to elaborate on "extremely crappy"? Is this where we either rewrite everything to use a different storage api, or just give up and let legacy be legacy?

12 rats tied together posted:

e: If you're lucky you can just delete your AD domain too, while you're at it
We have multiple forests :eng99:

xzzy
Mar 5, 2009

An on site kube cluster is a full time job for at least one person. It constantly needs TLC and the learning curve is pretty steep too.

So hopefully your new VP is prepared to pay for that.

12 rats tied together
Sep 7, 2006

The NPC posted:

Care to elaborate on "extremely crappy"? Is this where we either rewrite everything to use a different storage api, or just give up and let legacy be legacy?

Basically it's the same functionality you would get from managing non-domain credentials with your choice of app orchestration tooling created any time in the past 20 years, except it is an express trip to rat hole hell if you try to integrate it with anything that isn't the rest of your licensed microsoft ecosystem.

If you're leaning hard into k8s I would absolutely rewrite for a different storage api. Basically just browse through the volume mappings in the api spec and pick one that sounds good, but you can't really go wrong with NFS. If you're running this in a cloud provider there is likely already support for your vendor's version of managed NFS (ex: AWS EFS).

Depending on your access patterns there are possibly even better/simpler solutions. If this is just config reads, for example, you can drop those into a configmap and be done with it. If this is data that is read-on-startup, you can throw it into a network cache like redis or memcached.

Write-only workflows you're probably better served by a log pipeline or by some type of event store/event bus. Read/write should probably be pointed at a database instead of a file system. These are all hard to change for legacy software of course.

Zephirus
May 18, 2004

BRRRR......CHK
There are CSI drivers for AzureFile, which is essentially a SMB3 share, and there are existing custom drivers for it in Azure AKS, however both of these are limited to actual azure file resources and don't support custom cifs shares.

Also echoing all the other comments about windows containers, it's not good.

You will have to track your windows container bases with the version on your windows nodes as there's no hyper-v isolation, so you have to run all the containers in process isolation mode (and you can't do this with mismatched os versions). When/if you upgrade your nodes, you'll have to be very careful with scheduling to avoid it, usually create a new node pool and ensure new containers built are deployed onto that, which you'll have to do every 6 months or so if you're on SAC.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

The NPC posted:

:shrug: We have a new VP who wants to build in-house PaaS offerings for all of our Ops services.

You don't need to use containers or Kubernetes to do that. This sounds like the typical case of the new, semi-technical leadership making sweeping proclamations about what the engineering side of the company must do going forward with no understanding of the current state or what it would take to implement their proclamation.

Kubernetes exists to solve a problem. It solves that problem by introducing new, different problems. Such is the way of all technology since the dawn of time. If you're not having a challenge with your application that Kubernetes is designed to solve, moving it into Kubernetes will retain the existing problems while adding additional problems.

Hadlock
Nov 9, 2004

The NPC posted:

:shrug: We have a new VP who wants to build in-house PaaS offerings for all of our Ops services running windows k8s on bare metal

Update your resume and leave

Methanar
Sep 26, 2013

by the sex ghost

The NPC posted:

:shrug: We have a new VP who wants to build in-house PaaS offerings for all of our Ops services.

What does this have to do with assigning containers to VMs.

12 rats tied together
Sep 7, 2006

I'm not going to interrogate this person because they have a dumb VP but I just want to echo that yeah, containers or vm or native is purely an ops concern.

You should be suspicious once development starts to care about where their executables run because it's likely that problem fit has been thrown out the window in favor of resume development. You can run SOA, or even a full microservice stack, in any language, any tech stack, and on top of any platform. Both are application design paradigms and have nothing to do with where or how the code runs.

Adbot
ADBOT LOVES YOU

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

12 rats tied together posted:

purely an ops concern.

That's a very anti-devops attitude that I disagree with in every way. We should be trying to eliminate silos, not saying "gently caress it, that's an ops problem. I threw my code over the wall to them, let them figure out what to do with it now".

[edit]
In fact, you're seeing evidence right now of precisely how it isn't an ops problem: The application wasn't engineered with platform portability in mind. Keeping an attitude of "it's an ops problem" means that no engineering effort will ever be spent in solving that problem.

New Yorp New Yorp fucked around with this message at 21:47 on Mar 27, 2021

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply