|
12 rats tied together posted:- every business unit making technology choices meets the requirements of supporting that technology choice for, at least, their own planned use of it If I hear about something that sounds questionable, I will absolutely flag the risk to whomever has the job of managing that risk. Beyond that, a person insistent on keeping a decision that's been flagged to them as risky is firmly in "gently caress around and find out" territory. It's not my job to specify the precise ways that a specific alternative technology implementation adds sprawl to the company, or to imagine the world ten steps out where this tech experiment has run its course and now needs to be migrated. Every engineering leader in the company should already be thinking about this and having a perspective on when that risk is or isn't okay to incur. So many companies run on the backs of this kind of free labor from random people across the company. I'll almost always do this anyway for the people who are genuinely curious after having the risk flagged, and want to have a conversation to learn more. I'll never again write up the full executive summary to send into the wind, where nobody will read it, when that isn't related whatsoever to the impact that I'm accountable for delivering. Systems thinkers are in the unfortunate position of seeing our role in every problem. It makes it very hard to have good boundaries. However, companies cease to function effectively when they have too many people without good boundaries. 12 rats tied together posted:every staff member that reacts to someone else's technology choice does so by pivoting, instead of having it be explicitly their job So often, these things happen because a company has incoherent or poorly-bought-in views on team autonomy vs. central technical governance. People pay lip service to the idea of technical review, but then leadership is never accountable for ensuring that teams' technical decisions actually flow through the process, and nobody's invested in making it work correctly. Nothing good ever comes from prolonging this state of affairs, or from giving people an "out" so they can avoid ever having to confront it. Heroics might improve an individual outcome but are likely to actually undermine the function and resilience of the system in the long term. Gatekeeping can be good, because some changes are actually very risky. Trying to thread this needle has been the key challenge of infosec as a discipline for decades. It's absolutely crucial that everybody actually recognizes this and gets aligned on what the thought process and compromises are supposed to look like for incorporating that risk calculus into decision-making. When people have to do this ad-hoc, the incentives go sideways. The involvement of an influential person saying or not saying "I don't think you should go ahead with this idea" seems capricious and arbitrary and political. People earn individual reputations as gatekeepers and individually have to discuss their personal ideas of risk with each person in the company trying to move a technology decision forward. Failing to operationalize this leads to a really lovely state of affairs across the entire organization. 12 rats tied together posted:I also don't think that every "strategic investment" in a particular technology choice is done out of malice or ignorance, I do believe that it is possible to create healthy platform ownership teams, and we are much more firmly back into "it depends" territory in the presence of either of these things. Vulture Culture fucked around with this message at 04:56 on Feb 23, 2022 |
# ? Feb 23, 2022 04:52 |
|
|
# ? Jun 5, 2024 04:33 |
|
Anyone want to talk me into / talk me out of traefik? It feels like it's become a pretty common solution for routing requests to services, but I'm really suspicious of it requiring admin privileges to set up a CRD for ingress. Doesn't that give it a lot of freedom to screw with routes (and potentially screw things up)? I've been using nginx with manually configured reverse proxies to this point but am working on a project where that might not be the best idea anymore (too many proxies). But I really like not needing any elevated access too.
|
# ? Feb 23, 2022 22:35 |
|
I've used it in production, never gave me any headaches, has a couple of cool features, seems as reliable as nginx Nobody ever got fired for deploying nginx in prod though
|
# ? Feb 24, 2022 03:14 |
|
Traefik is sweet. The docs are sometimes lacking, and the label system is somewhat funky, but it's really flexible and performant as hell.
|
# ? Feb 24, 2022 03:27 |
|
xzzy posted:Anyone want to talk me into / talk me out of traefik? It feels like it's become a pretty common solution for routing requests to services, but I'm really suspicious of it requiring admin privileges to set up a CRD for ingress. Doesn't that give it a lot of freedom to screw with routes (and potentially screw things up)? As with any automation with regards to ingress it has the potential to mess up your routing big time. This is why you run tests though. Traefik works nicely for this but so does haproxy or nginx. What makes you choose Traefik specifically?
|
# ? Feb 24, 2022 10:46 |
|
LochNessMonster posted:What makes you choose Traefik specifically? Only that I see it suggested a lot, and auto config of reverse proxies sounds quite convenient. So I'm exploring it as an option but the CRD is a big red flag for me.
|
# ? Feb 24, 2022 16:02 |
|
caveat: knowledgeable about K8s in general but have never used Traefik specificallyxzzy posted:Only that I see it suggested a lot, and auto config of reverse proxies sounds quite convenient. xzzy posted:So I'm exploring it as an option but the CRD is a big red flag for me. b) If it really makes you uncomfortable, Traefik still works fine using the Kubernetes Ingress controller, it's just that the annotation-based syntax is pretty cumbersome and may not encapsulate all the features that the Traefik IngressRoute does
|
# ? Feb 24, 2022 16:41 |
|
Where do you see Traefik requiring cluster admin? I'm guessing you are following an implementation guide of some sort, but the RBAC in the helm chart looks fairly non-threatening... https://github.com/traefik/traefik-helm-chart/blob/master/traefik/templates/rbac/clusterrole.yaml That said, unless you are doing something particularly nifty, probably just use nginx as an ingress-controller Junkiebev fucked around with this message at 20:37 on Feb 25, 2022 |
# ? Feb 25, 2022 20:33 |
|
Hey if there’s a better docker thread I apologize but I think this might be the right place. Is there a way to get a windows container to feed a certain app’s Win EventLog entries to docker’s standard logging provider? Which, I mean, I think is just stdout/stderr? I got tired of running a bespoke VM for this tiny old hardware emulator app I use every now and then. It’s a 32bit windows service but I can make it run just the same by launching the binary as the entry point. The catch is that the only thing it dumps to stdout is “oh hey I started” and it writes its actual useful output as windows events. So I’d love to be able to interrogate the container for its logs without having to exec in with powershell to run get-winevent or whatever it is, I forgot. I’m fairly green to containers so if this is obvious I apologize. Logically in my head this would work well if I could figure out a way to start the server process but have a separate infinite loop that timestamps, sleeps for 2 seconds and then just pulls all events for that -ProviderName since the last timestamp. But I’m not really sure how I’d accomplish starting a task in the background with a Dockerfile on Windows, let alone setting up two separate things running concurrently. Anyway, just curious if there’s any feedback. Environment is plain docker on a POC windows 2022 server I’m eval for this. This isn’t really the IDEAL app for containerizing — it’s fairly hands off when all you want to do is interrogate the fake device, but to make config changes you still need to exec into the container to modify configs or have it generate keys or something. There’s probably good container hygiene I can apply to this but I haven’t really thought it through yet. At any rate, it’s a good excuse to learn about the tech even if it’s not the best use of it.
|
# ? Mar 4, 2022 05:23 |
|
Are you actually running windows containers in production
|
# ? Mar 4, 2022 14:27 |
|
this is at home -- we do a lot of dumb things at work but not THAT ballsy Also looks like MS already has me covered if I would just rtfm a little: https://github.com/microsoft/windows-container-tools/tree/master/LogMonitor This isn't anything critical. At work I have the actual hardware to ping against but I can't spare the 20k to buy my own some kinda jackal fucked around with this message at 14:47 on Mar 4, 2022 |
# ? Mar 4, 2022 14:45 |
|
Hadlock posted:Are you actually running windows containers in production Not to original poster to the question, and not currently but I did some consulting for a pretty large name running Windows containers in production. Hint: it's a shitshow and dreadful to work with at scale and I did not accept any follow-on work after the initial engagement
|
# ? Mar 4, 2022 14:49 |
|
Well that brings the number of times i've ever read about someone using windows containers in production to exactly one
|
# ? Mar 4, 2022 15:22 |
|
We're using windows containers in production. It's for build executors, so only internal I guess.
|
# ? Mar 4, 2022 15:45 |
I once POC’d Windows SQL containers for a client that insisted on it and it was just as stupid as you could imagine I thought Windows containers were dead. I learned something today
|
|
# ? Mar 4, 2022 15:51 |
|
And just now I kludged my stupid cli app to run under wine in a linux container like god intended so my reason to muck with Windows containers has been reduced to zero.
|
# ? Mar 4, 2022 17:43 |
|
we are running kubernetes on containerd windows 2022, authing via GMSA, in production
Junkiebev fucked around with this message at 23:59 on Mar 4, 2022 |
# ? Mar 4, 2022 22:28 |
|
I like Drone. I do not like being the only person in this 200 person company that understands or even knows about "my" EC2 Drone cluster. I am converting my Drone pipelines to GitHub Actions. I do not like GitHub Actions.
|
# ? Mar 5, 2022 04:40 |
|
Just curious, how do windows licenses work for containers?
|
# ? Mar 5, 2022 11:07 |
|
Windows Server has a 180 day trial, just spin the container up again before that.
|
# ? Mar 5, 2022 14:06 |
I remember that being kinda unclear and earnestly believing MS doesn’t care because using Windows containers was so awful they’d never make any money off of it
|
|
# ? Mar 5, 2022 14:38 |
|
LochNessMonster posted:Just curious, how do windows licenses work for containers? It’s quite murky! The ad-joined host talks to volume licensing servers, but as for the pods?
|
# ? Mar 5, 2022 16:48 |
|
Is there a tool for beautifying terraform hcl? I’m inheriting a dog’s breakfast with inconsistent *everything* and would prefer not to have to rewrite a bunch of it so as to be legible
|
# ? Mar 5, 2022 16:53 |
|
Junkiebev posted:Is there a tool for beautifying terraform hcl? I’m inheriting a dog’s breakfast with inconsistent *everything* and would prefer not to have to rewrite a bunch of it so as to be legible hclfmt is around: https://github.com/fatih/hclfmt however, the generally used version (above) of the tool has a major bug with consecutively commented lines and also was abandoned (but mostly works fine) I found, buried in one of the hashi repos, they seem to have either rewritten or forked the tool above (tbh I didnt really look closely), but you gotta compile yourself: https://github.com/hashicorp/hcl/tree/main/cmd/hclfmt this version fixes the issues I had Walked fucked around with this message at 17:07 on Mar 5, 2022 |
# ? Mar 5, 2022 17:03 |
|
Walked posted:hclfmt is around: https://github.com/fatih/hclfmt Thanks!
|
# ? Mar 5, 2022 17:38 |
|
Junkiebev posted:Is there a tool for beautifying terraform hcl? I’m inheriting a dog’s breakfast with inconsistent *everything* and would prefer not to have to rewrite a bunch of it so as to be legible Built in fmt command? I’ve had good results with it. https://www.terraform.io/cli/commands/fmt
|
# ? Mar 6, 2022 14:55 |
|
Vulture Culture posted:I agree, a lot of people make bad decisions [...] Thanks, I didn't have the time available to focus on reading this entire thing and considering it meaningfully until now, but its a good post and I appreciate it. I have been purely a line-level IC until recently and I appreciate your insight in this matter because I recognize you from other threads and have respect for your experience. I want to poke fun briefly though that your post contains an alarmingly high amount of "should", and while I disagree with none of it, it is not reality for me. It would be really nice though. I do agree that I was a little thoughtless with this statement: 12 rats tied together posted:Which services, components, facets, or whatever of your employer's technology footprint are best owned vs best collaboratively understood varies depending on way too many factors to make any strong assertions about. I think there are a bunch of really good and intuitive "rules of thumb" that we all mostly agree on. The largest and most interesting factor I concern myself with is the productivity of my customers, for example, it's a low-leverage use of time for a sr. frontend engineer to be learning all of the ways Terraform can be bad. I'd go as far as suggesting that docker and kubernetes are not developer concerns, they're purely ops domain technology choices that are best fit for certain deployment models that IC developers generally should not worry about. A sufficiently robust operations division ideally provides infrastructure abstractions, the way that is orchestrated in reality is an implementation detail. It's not particularly challenging or interesting to run a "dockerized" service without containers, or to "dockerize" a jar, for instance, and we all agree that kubernetes, a public cloud, and virtual machines do not solve for every possible application need. I think it naturally follows, then, that infrastructure should be owned by infrastructure, feature owned by feature, and that the original OP was right to put themselves in front of the clearly arbitrary nomad deployment choice, not that we ever disagreed at least on this aspect of it.
|
# ? Mar 8, 2022 20:48 |
|
I'm trying to create a basic Jenkins pipeline for a python project at work and it's going nowhere. I've already reached out for help and nobody has a clue what's up, myself included. The python project uses poetry as the package manager. The problem is that the Jenkins pipeline hangs on the first execution of an "sh" step. I'm new to CI/CD. Here is the Dockerfile: code:
code:
America Inc. fucked around with this message at 01:50 on Mar 11, 2022 |
# ? Mar 11, 2022 01:48 |
|
Maybe try the sh() fucntion to get any output the command is producing? Something like this:code:
I known absolutely nothing about poetry so can only give general suggestions.
|
# ? Mar 11, 2022 02:25 |
|
I did that and the pipeline hung again, and the console output showed this error:code:
|
# ? Mar 11, 2022 03:34 |
|
That error sounds like there’s a misconfiguration of the Jenkins builder node rather than something’s wrong with the Dockerfile. You don’t happen to have direct access to the Jenkins builder node do you?
|
# ? Mar 11, 2022 04:23 |
|
necrobobsledder posted:That error sounds like there’s a misconfiguration of the Jenkins builder node rather than something’s wrong with the Dockerfile. You don’t happen to have direct access to the Jenkins builder node do you? No, the node belongs to a coworker. That is making me think though, I could create my own EC2 instance and debug from it. For instance, I could put a `input` statement in the Jenkins pipeline to pause it, connect to the EC2 instance, connect to the detached docker image, and then see what's going on.
|
# ? Mar 11, 2022 06:46 |
|
I could be wrong in my approach, but I’d suggest trying to replicate the process in a “normal” non container node just to rule out any weirdness with the container. They’re great for ephemeral setups but the addd layer of fun can make things a bit hinky at times. Once you’ve confirmed/denied that’s part of the issue then you can scope down what you have to consider for troubleshooting.
|
# ? Mar 11, 2022 14:47 |
|
Look into interactive (bashrc) vs non interactive variables, for starters. We use poetry but install via asdf-vm. Poetry cache is also funky, we typically do poetry install at the beginning of the pipeline. Depending on when you run that again, poetry might be unable to write to the cache location. Either way echoing what other goons said. I'd pull jenkins-agent and build the script steps there and make sure they run, then implement as the Jenkins pipeline. One last poetry thing is that if you're invoking anything python when using it you probably want to be doing it with poetry run foo to make sure you use it's virtual env and path.
|
# ? Mar 12, 2022 03:26 |
|
It turns out the problem was with the builder node; I changed to a different one and the problem went away. I do have to say though, coming from more of a frontend background I find Jenkins to be often non-intuitive. And then there's these weird kludges, like how docker images are run like code:
Thanks for the help.
|
# ? Mar 12, 2022 03:54 |
|
Jenkins is just a victim of its age and userbase. There are so many ways to do anything and so many developers willing to write new plugins that it can be a giant pain to sift through documentation. Jenkinsfiles did help a lot IMO. But I'll agree their integration with the container universe has been less than ideal.
|
# ? Mar 12, 2022 04:01 |
|
Jenkins is the model t of modern ci/cd There's been a lot of improvements since then, but the technology is so well understood and durable that you still see them tooling around in the modern day Our Jenkins instances are more reliable than circle ci, so there's that, too. We have a Jenkins notification to tell us when circle ci is down, because otherwise people complain that their builds aren't working...
|
# ? Mar 12, 2022 04:15 |
|
About Jenkins - this book, at least from its Table of Contents, seems like a good resource: https://www.manning.com/books/pipeline-as-code?query=Jenkins
|
# ? Mar 12, 2022 05:37 |
|
I'm also a Stan for Jenkins, my teams tend to build the manager image using config as code and pinned plugin versions. We use Jenkins job builder and Jenkins files (w yaml support) and most of our teams are happy. If we could run everything where I'm at now in k8s I'd probably go all in on gitlab or similar just for the reduced ops overhead.
|
# ? Mar 12, 2022 22:06 |
|
|
# ? Jun 5, 2024 04:33 |
|
"can you grant this dev user access to important_prod_s3_bucket?" me: "absolutely not, that bucket has PII" "why not?" There's like an entire group of 20 developers siloed in another part of the company and they're all like this. I'm genuinely afraid to look under the covers of what the gently caress they're doing over there, guessing it looks like a late 90s windows shop over there or something They have a managed k8s cluster but still run all their services on dedicated EC2 instances and manually deploy new versions by hand Edit
|
# ? Mar 14, 2022 18:10 |