Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

12 rats tied together posted:

- every business unit making technology choices meets the requirements of supporting that technology choice for, at least, their own planned use of it
I agree, a lot of people make bad decisions to use technology that they're unable to support. But, from my perspective: as a technologist, I've got a job to do. If I help to set a precedent that I, and my fellow platform team technologists, have the additional (uncompensated) job of tapping into the feed of upcoming technology decisions, vetting them to make sure they're all good choices, and providing concierge service explaining to people why their decisions are sub-optimal, I'm subverting a few areas where the company really needs to develop mature models and processes in order to be successful.

If I hear about something that sounds questionable, I will absolutely flag the risk to whomever has the job of managing that risk. Beyond that, a person insistent on keeping a decision that's been flagged to them as risky is firmly in "gently caress around and find out" territory. It's not my job to specify the precise ways that a specific alternative technology implementation adds sprawl to the company, or to imagine the world ten steps out where this tech experiment has run its course and now needs to be migrated. Every engineering leader in the company should already be thinking about this and having a perspective on when that risk is or isn't okay to incur. So many companies run on the backs of this kind of free labor from random people across the company. I'll almost always do this anyway for the people who are genuinely curious after having the risk flagged, and want to have a conversation to learn more. I'll never again write up the full executive summary to send into the wind, where nobody will read it, when that isn't related whatsoever to the impact that I'm accountable for delivering.

Systems thinkers are in the unfortunate position of seeing our role in every problem. It makes it very hard to have good boundaries. However, companies cease to function effectively when they have too many people without good boundaries.

12 rats tied together posted:

every staff member that reacts to someone else's technology choice does so by pivoting, instead of having it be explicitly their job
This still shouldn't be reactive, it should be thoughtful and deliberate. If you're in an environment where there are significant consequences at scale of having diverging technology choices throughout the company, there should be a process for vetting new technologies. Some companies have a technical governance committee steered by engineering leadership, others have an architecture review board run by staff individual contributors or full-time architects. Whatever the process is, it should be followed, and there should never be a situation where someone (especially a random, if highly-skilled, individual contributor!) is having a snap conversation to get out in front of a technical decision because it didn't flow through the right channels.

So often, these things happen because a company has incoherent or poorly-bought-in views on team autonomy vs. central technical governance. People pay lip service to the idea of technical review, but then leadership is never accountable for ensuring that teams' technical decisions actually flow through the process, and nobody's invested in making it work correctly. Nothing good ever comes from prolonging this state of affairs, or from giving people an "out" so they can avoid ever having to confront it. Heroics might improve an individual outcome but are likely to actually undermine the function and resilience of the system in the long term.

Gatekeeping can be good, because some changes are actually very risky. Trying to thread this needle has been the key challenge of infosec as a discipline for decades. It's absolutely crucial that everybody actually recognizes this and gets aligned on what the thought process and compromises are supposed to look like for incorporating that risk calculus into decision-making. When people have to do this ad-hoc, the incentives go sideways. The involvement of an influential person saying or not saying "I don't think you should go ahead with this idea" seems capricious and arbitrary and political. People earn individual reputations as gatekeepers and individually have to discuss their personal ideas of risk with each person in the company trying to move a technology decision forward. Failing to operationalize this leads to a really lovely state of affairs across the entire organization.

12 rats tied together posted:

I also don't think that every "strategic investment" in a particular technology choice is done out of malice or ignorance, I do believe that it is possible to create healthy platform ownership teams, and we are much more firmly back into "it depends" territory in the presence of either of these things.
Internal platforms are really important. I don't think any business that can survive without them is actually making or doing anything new. However, it's not an engineer's job (below Principal level, anyway) to take on ad-hoc, bottom-up work across the entire enterprise to make sure an engineering organization has a healthy and coherent platform strategy. It is every engineer's job to communicate their strategic intent, and notify other people/teams around them of where they envision their scope changing. If you imagine that a platform you manage will progress from being an opt-in platform, to being an opt-out platform, to being a mandatory platform, say so (or have someone in leadership message that on your behalf). Enable people to have the same context, because if you line up context, values, and incentives, you end up with people making decisions that converge to the same place. When an engineer creates surprises, and astonishes someone by randomly asserting their platform/product/team's scope is bigger than anyone previously believed it to be, and that there's a murky unforeseen consequence for not moving in lock-step with that team, that behavior is going to start adding friction and creating problems for everyone.

Vulture Culture fucked around with this message at 04:56 on Feb 23, 2022

Adbot
ADBOT LOVES YOU

xzzy
Mar 5, 2009

Anyone want to talk me into / talk me out of traefik? It feels like it's become a pretty common solution for routing requests to services, but I'm really suspicious of it requiring admin privileges to set up a CRD for ingress. Doesn't that give it a lot of freedom to screw with routes (and potentially screw things up)?

I've been using nginx with manually configured reverse proxies to this point but am working on a project where that might not be the best idea anymore (too many proxies). But I really like not needing any elevated access too.

Hadlock
Nov 9, 2004

I've used it in production, never gave me any headaches, has a couple of cool features, seems as reliable as nginx

Nobody ever got fired for deploying nginx in prod though

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.
Traefik is sweet. The docs are sometimes lacking, and the label system is somewhat funky, but it's really flexible and performant as hell.

LochNessMonster
Feb 3, 2005

I need about three fitty


xzzy posted:

Anyone want to talk me into / talk me out of traefik? It feels like it's become a pretty common solution for routing requests to services, but I'm really suspicious of it requiring admin privileges to set up a CRD for ingress. Doesn't that give it a lot of freedom to screw with routes (and potentially screw things up)?

I've been using nginx with manually configured reverse proxies to this point but am working on a project where that might not be the best idea anymore (too many proxies). But I really like not needing any elevated access too.

As with any automation with regards to ingress it has the potential to mess up your routing big time. This is why you run tests though.

Traefik works nicely for this but so does haproxy or nginx. What makes you choose Traefik specifically?

xzzy
Mar 5, 2009

LochNessMonster posted:

What makes you choose Traefik specifically?

Only that I see it suggested a lot, and auto config of reverse proxies sounds quite convenient.

So I'm exploring it as an option but the CRD is a big red flag for me.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.
caveat: knowledgeable about K8s in general but have never used Traefik specifically

xzzy posted:

Only that I see it suggested a lot, and auto config of reverse proxies sounds quite convenient.
"Auto config" in what sense? In broad terms, this is basically what all ingress controllers do.

xzzy posted:

So I'm exploring it as an option but the CRD is a big red flag for me.
a) You only need cluster-level permissions to install the CRD itself; once you have the CRD, all custom resources that are instances of your CRD classes can be managed through standard Kubernetes RBAC
b) If it really makes you uncomfortable, Traefik still works fine using the Kubernetes Ingress controller, it's just that the annotation-based syntax is pretty cumbersome and may not encapsulate all the features that the Traefik IngressRoute does

Junkiebev
Jan 18, 2002


Feel the progress.

Where do you see Traefik requiring cluster admin? I'm guessing you are following an implementation guide of some sort, but the RBAC in the helm chart looks fairly non-threatening...

https://github.com/traefik/traefik-helm-chart/blob/master/traefik/templates/rbac/clusterrole.yaml

That said, unless you are doing something particularly nifty, probably just use nginx as an ingress-controller

Junkiebev fucked around with this message at 20:37 on Feb 25, 2022

some kinda jackal
Feb 25, 2003

 
 
Hey if there’s a better docker thread I apologize but I think this might be the right place.

Is there a way to get a windows container to feed a certain app’s Win EventLog entries to docker’s standard logging provider? Which, I mean, I think is just stdout/stderr?

I got tired of running a bespoke VM for this tiny old hardware emulator app I use every now and then. It’s a 32bit windows service but I can make it run just the same by launching the binary as the entry point. The catch is that the only thing it dumps to stdout is “oh hey I started” and it writes its actual useful output as windows events.

So I’d love to be able to interrogate the container for its logs without having to exec in with powershell to run get-winevent or whatever it is, I forgot.

I’m fairly green to containers so if this is obvious I apologize. Logically in my head this would work well if I could figure out a way to start the server process but have a separate infinite loop that timestamps, sleeps for 2 seconds and then just pulls all events for that -ProviderName since the last timestamp. But I’m not really sure how I’d accomplish starting a task in the background with a Dockerfile on Windows, let alone setting up two separate things running concurrently.

Anyway, just curious if there’s any feedback. Environment is plain docker on a POC windows 2022 server I’m eval for this.


This isn’t really the IDEAL app for containerizing — it’s fairly hands off when all you want to do is interrogate the fake device, but to make config changes you still need to exec into the container to modify configs or have it generate keys or something. There’s probably good container hygiene I can apply to this but I haven’t really thought it through yet. At any rate, it’s a good excuse to learn about the tech even if it’s not the best use of it.

Hadlock
Nov 9, 2004

Are you actually running windows containers in production

some kinda jackal
Feb 25, 2003

 
 
:lol: this is at home -- we do a lot of dumb things at work but not THAT ballsy

Also looks like MS already has me covered if I would just rtfm a little: https://github.com/microsoft/windows-container-tools/tree/master/LogMonitor

This isn't anything critical. At work I have the actual hardware to ping against but I can't spare the 20k to buy my own :(

some kinda jackal fucked around with this message at 14:47 on Mar 4, 2022

Walked
Apr 14, 2003

Hadlock posted:

Are you actually running windows containers in production

Not to original poster to the question, and not currently but I did some consulting for a pretty large name running Windows containers in production.

Hint: it's a shitshow and dreadful to work with at scale and I did not accept any follow-on work after the initial engagement

Hadlock
Nov 9, 2004

Well that brings the number of times i've ever read about someone using windows containers in production to exactly one

PerOlus
Jan 26, 2003

We'r even, señor!
We're using windows containers in production. :shobon:
It's for build executors, so only internal I guess.

i am a moron
Nov 12, 2020

"I think if there’s one thing we can all agree on it’s that Penn State and Michigan both suck and are garbage and it’s hilarious Michigan fans are freaking out thinking this is their natty window when they can’t even beat a B12 team in the playoffs lmao"
I once POC’d Windows SQL containers for a client that insisted on it and it was just as stupid as you could imagine

I thought Windows containers were dead. I learned something today

some kinda jackal
Feb 25, 2003

 
 
And just now I kludged my stupid cli app to run under wine in a linux container like god intended so my reason to muck with Windows containers has been reduced to zero.

Junkiebev
Jan 18, 2002


Feel the progress.

we are running kubernetes on containerd windows 2022, authing via GMSA, in production :getin:

Junkiebev fucked around with this message at 23:59 on Mar 4, 2022

pentium166
Oct 15, 2012
I like Drone.
I do not like being the only person in this 200 person company that understands or even knows about "my" EC2 Drone cluster.
I am converting my Drone pipelines to GitHub Actions.


I do not like GitHub Actions.

LochNessMonster
Feb 3, 2005

I need about three fitty


Just curious, how do windows licenses work for containers?

Mustache Ride
Sep 11, 2001



Windows Server has a 180 day trial, just spin the container up again before that.

i am a moron
Nov 12, 2020

"I think if there’s one thing we can all agree on it’s that Penn State and Michigan both suck and are garbage and it’s hilarious Michigan fans are freaking out thinking this is their natty window when they can’t even beat a B12 team in the playoffs lmao"
I remember that being kinda unclear and earnestly believing MS doesn’t care because using Windows containers was so awful they’d never make any money off of it

Junkiebev
Jan 18, 2002


Feel the progress.

LochNessMonster posted:

Just curious, how do windows licenses work for containers?

It’s quite murky! The ad-joined host talks to volume licensing servers, but as for the pods? :iiam:

Junkiebev
Jan 18, 2002


Feel the progress.

Is there a tool for beautifying terraform hcl? I’m inheriting a dog’s breakfast with inconsistent *everything* and would prefer not to have to rewrite a bunch of it so as to be legible

Walked
Apr 14, 2003

Junkiebev posted:

Is there a tool for beautifying terraform hcl? I’m inheriting a dog’s breakfast with inconsistent *everything* and would prefer not to have to rewrite a bunch of it so as to be legible

hclfmt is around: https://github.com/fatih/hclfmt

however, the generally used version (above) of the tool has a major bug with consecutively commented lines and also was abandoned (but mostly works fine)

I found, buried in one of the hashi repos, they seem to have either rewritten or forked the tool above (tbh I didnt really look closely), but you gotta compile yourself:
https://github.com/hashicorp/hcl/tree/main/cmd/hclfmt

this version fixes the issues I had

Walked fucked around with this message at 17:07 on Mar 5, 2022

Junkiebev
Jan 18, 2002


Feel the progress.

Walked posted:

hclfmt is around: https://github.com/fatih/hclfmt

however, the generally used version (above) of the tool has a major bug with consecutively commented lines and also was abandoned (but mostly works fine)

I found, buried in one of the hashi repos, they seem to have either rewritten or forked the tool above (tbh I didnt really look closely), but you gotta compile yourself:
https://github.com/hashicorp/hcl/tree/main/cmd/hclfmt

this version fixes the issues I had

Thanks!

SuitcasePimp
Feb 27, 2005

Junkiebev posted:

Is there a tool for beautifying terraform hcl? I’m inheriting a dog’s breakfast with inconsistent *everything* and would prefer not to have to rewrite a bunch of it so as to be legible

Built in fmt command? I’ve had good results with it.

https://www.terraform.io/cli/commands/fmt

12 rats tied together
Sep 7, 2006

Vulture Culture posted:

I agree, a lot of people make bad decisions [...]

Thanks, I didn't have the time available to focus on reading this entire thing and considering it meaningfully until now, but its a good post and I appreciate it. I have been purely a line-level IC until recently and I appreciate your insight in this matter because I recognize you from other threads and have respect for your experience.

I want to poke fun briefly though that your post contains an alarmingly high amount of "should", and while I disagree with none of it, it is not reality for me. :) It would be really nice though.

I do agree that I was a little thoughtless with this statement:

12 rats tied together posted:

Which services, components, facets, or whatever of your employer's technology footprint are best owned vs best collaboratively understood varies depending on way too many factors to make any strong assertions about.

I think there are a bunch of really good and intuitive "rules of thumb" that we all mostly agree on. The largest and most interesting factor I concern myself with is the productivity of my customers, for example, it's a low-leverage use of time for a sr. frontend engineer to be learning all of the ways Terraform can be bad.

I'd go as far as suggesting that docker and kubernetes are not developer concerns, they're purely ops domain technology choices that are best fit for certain deployment models that IC developers generally should not worry about. A sufficiently robust operations division ideally provides infrastructure abstractions, the way that is orchestrated in reality is an implementation detail. It's not particularly challenging or interesting to run a "dockerized" service without containers, or to "dockerize" a jar, for instance, and we all agree that kubernetes, a public cloud, and virtual machines do not solve for every possible application need.

I think it naturally follows, then, that infrastructure should be owned by infrastructure, feature owned by feature, and that the original OP was right to put themselves in front of the clearly arbitrary nomad deployment choice, not that we ever disagreed at least on this aspect of it.

America Inc.
Nov 22, 2013

I plan to live forever, of course, but barring that I'd settle for a couple thousand years. Even 500 would be pretty nice.
I'm trying to create a basic Jenkins pipeline for a python project at work and it's going nowhere. I've already reached out for help and nobody has a clue what's up, myself included. The python project uses poetry as the package manager. The problem is that the Jenkins pipeline hangs on the first execution of an "sh" step. I'm new to CI/CD.

Here is the Dockerfile:
code:
FROM python:3.7.12-buster

# Install dependencies.
RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
RUN apt-get install -y lftp git git-lfs gcc python3.7-venv
RUN git lfs install

ENV WORKSPACE /workspace
ENV HOME /home/jenkins
RUN mkdir -p $WORKSPACE

# Creating user jenkins
RUN adduser --uid 1001 --shell /bin/bash --home $HOME jenkins
RUN chown jenkins:jenkins $WORKSPACE

USER jenkins

WORKDIR $WORKSPACE

# Copy repo into workspace folder.
ADD --chown=jenkins:jenkins . $WORKSPACE/

ARG POETRY_VERSION="1.1.13"
RUN pip3 install poetry==${POETRY_VERSION}

# Ensure poetry is in the PATH.
RUN echo "export PATH=$PATH:/home/jenkins/.local/bin" >> ~/.bashrc
CMD ["bash"]
Here is the Jenkinsfile (in declarative syntax):
code:
pipeline {
    // Building and running from the Dockerfile at the root of the repo.
    // The final version of this will of course pull a pre-built Docker image,
    // but it's easier to iterate this way.
    agent { 
        dockerfile {
            label "some ec2 node"
            args "-u jenkins:jenkins"
        }
    }

    stages {
        stage("Checkout") {
            steps {
                checkout scm
            }
        }
    
        stage("Poetry Configuration") {
            steps {
               // Hangs here.
                sh "poetry env use python3.7 && cd our-repo && poetry install --no-root"
            }
        }
    
        stage("Test some stuff") {
            steps {
                sh "cd our-repo && poetry run some-script.py"
            }
        }
    }
}
Can somebody give me some pointers of where to look here/bad practice? I have done a million replays fiddling with this and that.

America Inc. fucked around with this message at 01:50 on Mar 11, 2022

xzzy
Mar 5, 2009

Maybe try the sh() fucntion to get any output the command is producing? Something like this:

code:
    
stage('butts') {
      steps {
        script {
          buf = sh(
            script: '''
              echo butts
            ''',
            returnStdout: true
          ).trim()
          echo "${buf}"
        }
      }
    }
If you need the stderr, do the 2>&1 redirection.

I known absolutely nothing about poetry so can only give general suggestions. :v:

America Inc.
Nov 22, 2013

I plan to live forever, of course, but barring that I'd settle for a couple thousand years. Even 500 would be pretty nice.
I did that and the pipeline hung again, and the console output showed this error:
code:
process apparently never started in /home/ec2-user/workspace/[repo-my-branch]@tmp/durable-[random hash]
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true
might make the problem clearer)

Cannot contact [ec2 node]: java.io.FileNotFoundException:
/home/ec2-user/workspace/[repo-my-branch]@tmp/durable-[random hash]/output.txt (No such file or directory)
Note that I replaced sensitive info in brackets. I can try the LAUNCH_DIAGNOSTICS=true flag but I don't seem to have the permissions for that, which is a separate issue. I imagine that the output.txt file is probably the file that contains the output from script execution.

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
That error sounds like there’s a misconfiguration of the Jenkins builder node rather than something’s wrong with the Dockerfile. You don’t happen to have direct access to the Jenkins builder node do you?

America Inc.
Nov 22, 2013

I plan to live forever, of course, but barring that I'd settle for a couple thousand years. Even 500 would be pretty nice.

necrobobsledder posted:

That error sounds like there’s a misconfiguration of the Jenkins builder node rather than something’s wrong with the Dockerfile. You don’t happen to have direct access to the Jenkins builder node do you?

No, the node belongs to a coworker. That is making me think though, I could create my own EC2 instance and debug from it. For instance, I could put a `input` statement in the Jenkins pipeline to pause it, connect to the EC2 instance, connect to the detached docker image, and then see what's going on.

Warbird
May 23, 2012

America's Favorite Dumbass

I could be wrong in my approach, but I’d suggest trying to replicate the process in a “normal” non container node just to rule out any weirdness with the container. They’re great for ephemeral setups but the addd layer of fun can make things a bit hinky at times. Once you’ve confirmed/denied that’s part of the issue then you can scope down what you have to consider for troubleshooting.

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.
Look into interactive (bashrc) vs non interactive variables, for starters.

We use poetry but install via asdf-vm.

Poetry cache is also funky, we typically do poetry install at the beginning of the pipeline. Depending on when you run that again, poetry might be unable to write to the cache location.

Either way echoing what other goons said. I'd pull jenkins-agent and build the script steps there and make sure they run, then implement as the Jenkins pipeline.

One last poetry thing is that if you're invoking anything python when using it you probably want to be doing it with poetry run foo to make sure you use it's virtual env and path.

America Inc.
Nov 22, 2013

I plan to live forever, of course, but barring that I'd settle for a couple thousand years. Even 500 would be pretty nice.
It turns out the problem was with the builder node; I changed to a different one and the problem went away.

I do have to say though, coming from more of a frontend background I find Jenkins to be often non-intuitive. And then there's these weird kludges, like how docker images are run like
code:

docker run -d - t ... image_name cat
...like, why are they changing the CMD to cat? I found an explanation on stack overflow but running a docker image is such a common and basic thing to do for a CI/CD pipeline, right?

Thanks for the help.

xzzy
Mar 5, 2009

Jenkins is just a victim of its age and userbase. There are so many ways to do anything and so many developers willing to write new plugins that it can be a giant pain to sift through documentation.

Jenkinsfiles did help a lot IMO. But I'll agree their integration with the container universe has been less than ideal.

Hadlock
Nov 9, 2004

Jenkins is the model t of modern ci/cd

There's been a lot of improvements since then, but the technology is so well understood and durable that you still see them tooling around in the modern day

Our Jenkins instances are more reliable than circle ci, so there's that, too. We have a Jenkins notification to tell us when circle ci is down, because otherwise people complain that their builds aren't working...

Love Stole the Day
Nov 4, 2012
Please give me free quality professional advice so I can be a baby about it and insult you
About Jenkins - this book, at least from its Table of Contents, seems like a good resource: https://www.manning.com/books/pipeline-as-code?query=Jenkins

Gyshall
Feb 24, 2009

Had a couple of drinks.
Saw a couple of things.
I'm also a Stan for Jenkins, my teams tend to build the manager image using config as code and pinned plugin versions. We use Jenkins job builder and Jenkins files (w yaml support) and most of our teams are happy.

If we could run everything where I'm at now in k8s I'd probably go all in on gitlab or similar just for the reduced ops overhead.

Adbot
ADBOT LOVES YOU

Hadlock
Nov 9, 2004

:downs: "can you grant this dev user access to important_prod_s3_bucket?"
me: "absolutely not, that bucket has PII" :eng101:
:downs: "why not?"
:eng99:

There's like an entire group of 20 developers siloed in another part of the company and they're all like this. I'm genuinely afraid to look under the covers of what the gently caress they're doing over there, guessing it looks like a late 90s windows shop over there or something

They have a managed k8s cluster but still run all their services on dedicated EC2 instances and manually deploy new versions by hand

Edit

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply