|
The load balancer / reverse proxy should be pretty darn hardened and ideally its ssh / management ports would be inaccessible to basically everyone. Nginx is well known as a web server in this respect and is super secure (when properly setup) and will perform just fine with low latency in contrast to the built-in web servers of, say, Django or CouchDB. The Java ecosystem, in contrast, is all about embedding super heavyweight industrial-grade http servers like Tomcat into Spring and company (despite / because of their age / maturity) so it's a bit tougher to pull apart. A huge chunk of the leaked PII out there is people putting crap out on AWS S3 and getting frustrated with IAM's complexities to permit cross-account access to the buckets (something super common yet super awkward to do correctly). And AWS has the balls to put out Macie to charge people even more basically. Pretty amazing how Amazon can manage to charge you for 90%+ of the lifecycle of data and software alike now. necrobobsledder fucked around with this message at 13:53 on Apr 23, 2019 |
#
?
Apr 23, 2019 13:50
|
|
|
|
| # ? May 15, 2024 18:56 |
|
|
necrobobsledder posted:Pretty amazing how Amazon can manage to charge you for 90%+ of the lifecycle of data and software alike now.
|
|
#
?
Apr 23, 2019 20:48
|
|
|
Vulture Culture posted:Every tool wants to be a platform, and every company unknowingly wants to be IBM when it grows up Unknowingly?
|
|
#
?
Apr 23, 2019 21:11
|
|
|
NihilCredo posted:Unknowingly?
|
|
#
?
Apr 24, 2019 03:51
|
|
|
Sorry if this has already been answered, I've dug through the thread some and don't see anything recent that seems fairly specific. What do you guy's deployment pipelines look like? How do you go from source code in a repo to something running in k8s? I've been doing some research and I've been surprised by how little information I'm finding. Most of what I have been able to find is just trying to sell me a product. We're very early stages on getting clusters and the tooling around them setup. I'd be interested in hearing other's experience in this area, tips, tricks, pitfalls, etc.
|
|
#
?
Apr 30, 2019 05:03
|
|
|
PBS posted:Sorry if this has already been answered, I've dug through the thread some and don't see anything recent that seems fairly specific. A shell script wrapping helm upgrade --install to handle revert logic, sanity checking, inserting required variables regarding the environment in a Jenkins task is the best thing I've come up with so far. Then you can have your Jenkins trigger be pushes, merges or any other operation on a branch name pattern. Maybe like something like feature/* Everything is a lot easier if you don't have to handle things like draining external load balancers, extremely long lived sessions, or requirements regarding dealing with external services that must remain outside of the cluster, or anything like that. Methanar fucked around with this message at 05:17 on Apr 30, 2019 |
|
#
?
Apr 30, 2019 05:15
|
|
|
PBS posted:Sorry if this has already been answered, I've dug through the thread some and don't see anything recent that seems fairly specific. Build jobs produce docker images and helm charts, and bundle a deploy script. Deploy jobs consume that stuff. The build server doesn’t do anything that someone with docker, helm, kubectl, and bash installed couldn’t do. Some of the builds we converted to this structure were truly disgusting and flaky as hell, couldn’t survive outside their kludged together environment. We’re still grappling with deployment, mainly because the stuff we’re deploying, if it has any kind of health checking, has a weird hack different for each app that our old tooling knew how to deal with and decide whether a deploy was successful. We don’t deploy very often, so alerting on pods in CrashLoopBackOff state is an almost sufficient alternative to doing things properly, but liveness and readiness probes are real nice if you can use them.
|
|
#
?
Apr 30, 2019 05:39
|
|
|
Methanar posted:A shell script wrapping helm upgrade --install to handle revert logic, sanity checking, inserting required variables regarding the environment in a Jenkins task is the best thing I've come up with so far. Are you able to summarize what you believe helm offers over deploying via kubectl? I see some benefits like rollback and hooks but I'm short on bullet points in a pros/cons list. Bleeding will likely eventually be a concern but I don't see it being necessary with an external lb in the near future. We're a decent long lived fortune500 so we have a large number of existing services. Some of which would likely be moved into the k8s clusters, some would likely not. Things are fairly segregated here so a number of things will just never end up in our own cluster.
|
|
#
?
Apr 30, 2019 05:44
|
|
|
PBS posted:Are you able to summarize what you believe helm offers over deploying via kubectl? No matter what, you're really really going to want some kind of yaml templating preprocessing. Helm does that. But then again so does Kubectl natively as of somewhat recently since Kustomize got merged in. If I were to do my own stuff again from scratch I definitely probably go the Kustomize route. With bare Kubectl you may need a little bit more logic for things like inserting what docker tag you actually want to deploy is. Helm gives you an alrightish way of supplying variables at runtime, but I'm sure there is a way to do the same with just kubectl. Don't use any specific Jenkins plugins that use the word Kubernetes in them. Just use a shell thats executing a script wrapping helm, kubectl, etc. Also don't run Jenkins' agents in Kubernetes pods.
|
|
#
?
Apr 30, 2019 05:48
|
|
|
Helm gives you a really powerful, industry standard, official opinionated kubernetes way to do templating of deployments, also gives you access to external helm dependencies like other charts. Plus you can publish your charts to an internal chart server (I think the official one is still chartmuseum, which can use a number of backends including S3) so you have a single versioned source of truth Helm also has the option to output a set of files (rather than interact with the cluster directly) that you can then run kubectl against. This keeps things pretty modular and seems to be a good happy medium for a lot of teams. Helm 3.0 is supposed to drop support for tiller entirely which reduces a lot of the complexity and complaints people have about it (most/many tiller users are setup with root level privs). I would definitely consider using helm in some capacity, probably paired with a bash script or some (insert language of choice) it is a lot cleaner than manually maintaining deployments by hand and storing everything in source code. If you use helm secrets it's pretty straightforward to use charts with a secret store and not get your chocolate mixed up with your peanutbutter.
|
|
#
?
Apr 30, 2019 07:01
|
|
|
In the context of a project distributed as a Docker image (with a Dockerfile in the root directory), does it make sense to expose Prometheus metrics for the app itself ? The page I linked says it's possible to monitor a Docker instance using Prometheus. Knowing very little about Docker, I'm not sure what instance means in this context.
|
|
#
?
Apr 30, 2019 12:07
|
|
|
The docket daemon can be configured to show it's own metrics, just like you can turn on metrics for nginx and a whole host of other things. Turning this on exposes your docker daemon externally so that could likely be a security risk if you do so, but for internal testing it's occasionally useful for debug. I would hesitate to turn it on before you need it based on security concerns. In addition to that, many apps that you might run in containers, especially built for the cloud golang apps, have a /metrics endpoint that you can setup for Prometheus autodiscovery or manually confine to scrape. I generally expect to want to monitor every container that I launch and have it wire back to a grafana dashboard. Apps you write can also have Prometheus metrics added either manually or via a plugin. The link you use is referring to the first paragraph
|
|
#
?
Apr 30, 2019 13:10
|
|
|
Our deployment are like 20 'kubectl apply' commands in azure devops. Based on what we're doing currently with variable replacement for our yaml manifests in the pipeline, I don't really see the benefit for helm right now. I'm always open to change, but for us it just seems like another tool for the sake of tooling.
|
|
#
?
Apr 30, 2019 14:08
|
|
|
Spring Heeled Jack posted:Our deployment are like 20 'kubectl apply' commands in azure devops. Based on what we're doing currently with variable replacement for our yaml manifests in the pipeline, I don't really see the benefit for helm right now. I'm always open to change, but for us it just seems like another tool for the sake of tooling. Will somebody else on your team know how to run those exact 20 commands for your project? Helm makes deployment easy and uniform. And after learning from my first helm chart the second one was a breeze. Now I slap down my variables in the answer file and away the deployment goes. Anybody I work with can provide their own variables and deploy without having to know the inner workings of my app.
|
|
#
?
Apr 30, 2019 14:17
|
|
|
PBS posted:Sorry if this has already been answered, I've dug through the thread some and don't see anything recent that seems fairly specific. ansible playbook(s) that contain a series of k8s module calls that push j2 templated manifests to k8s. The j2 templating hooks into the rest of our ansible ecosystem which has hooks into and data about most of our infrastructure. The playbook steps are nice for managing ordering and dependencies of disparate k8s object applications, for example, push this thing and then watch this metric until it stabilizes, and then push the next thing, followed by the rest of the things in chunks of 20%. I also haven't needed to learn any of the 14 other types of k8s templating systems and my documentation search radius so far has been the k8s api documentation and k8s.py. I really like it. I do think one of the strengths of the k8s approach though is that you can pick whatever orchestration system you want. Templating text files and running shell commands in sequence has been trivial for about as long as computers have existed.
|
|
#
?
Apr 30, 2019 19:31
|
|
|
code:
|
|
#
?
Apr 30, 2019 21:59
|
|
|
Helm is the bomb. We use ansible for orchestrating everything else in an ephemeral Jenkins workspace.
|
|
#
?
May 1, 2019 00:53
|
|
|
12 rats tied together posted:ansible playbook(s) that contain a series of k8s module calls that push j2 templated manifests to k8s. I'll admit my eyes glazed over at this point but ansible to control k8s seems like some sort of 666 unholy antipattern regardless of how cool it is helm4lyfe
|
|
#
?
May 1, 2019 02:14
|
|
|
comedy answer: use ksonnet
|
|
#
?
May 1, 2019 02:39
|
|
|
We're using Rancher for orchestration. Having a questions file is nice for drop-down boxes and password fields instead of pasting everything in an answers file. But the helm caching really sucks (trying to load a new chart version can take multiple page refreshes), error handing is miserable (questions file malformed? Your chart will never show up, nor any versions past the problem chart), and bugs are abundant (deleting namespaces and resources doesn't always take, and they're left orphaned in k8s). I'm getting to the point where it's not worth the hassle anymore.
|
|
#
?
May 1, 2019 03:06
|
|
|
For those of us still unable to use any of that for deployments because you’re working with legacy applications that can’t be deployed in containers, the above posts show exactly what you can do by making your applications containerized. We’ve got several engineers working on tooling and helper programs for months and months just to be able to deploy the platform as a series of make targets with some decent level of safety. That’s also really expensive and takes a lot of time writing stuff that’s not that big of a competitive advantage to the business (seriously, when has “we have a unique deployment process that nobody else out there does” been a feature?). Being able to yum install your software is pretty familiar and tons of people know how it works. That’s one advantage of working on a mature product that’s been installed on-premise for years. The hard part is removing state all over the place usually, but the crazy thing is I’m on a system that’s already refactored to do single-box microservices and the primary state is just Postgres and Elasticsearch now with those both easy to externalize. So we’ll be able to go full retard K8S far before any other place I’ve worked that tried to lift and shift their pile of technical debt into a poorly laid out IaaS setup.
|
|
#
?
May 1, 2019 04:59
|
|
|
Hadlock posted:[...] ansible to control k8s seems like some sort of 666 unholy antipattern [...] It's not unreasonable to feel this way, but it helps if you remember that ansible is a deployment tool more than it is a configuration management tool. It's an extremely supported, common, and well known pattern to have ansible run tasks against its control machine. You don't have to leap very far from "use my control machine to make a series of AWS API calls" to land on "use my control machine to make a series of k8s API calls". It's been nice to not having to learn helm at all, but there's nothing stopping you from using ansible to orchestrate a series of helm commands either: https://docs.ansible.com/ansible/latest/modules/helm_module.html  Basically -- any time you have a readme, comment, wiki article, or otherwise that says "do this, then that", ansible is really good at handling that in a way that can be trivially understood and code reviewed.
|
|
#
?
May 1, 2019 15:02
|
|
|
After playing around with openshift, I think I’m in love.
|
|
#
?
May 1, 2019 15:28
|
|
|
ratbert90 posted:After playing around with openshift, I think I’m in love. Come back in a year and update this post
|
|
#
?
May 1, 2019 20:18
|
|
|
Gyshall posted:Come back in a year and update this post Same thing was said about Jenkins, and while we have moved on to gitlab-runner, I still quite like Jenkins and have no real issue with it. We switched to gitlab-runner for consolidation purposes.
|
|
#
?
May 2, 2019 04:01
|
|
|
Openshift is just a flavor of k8s as-a-service, right? What’s so potentially bad about it vs any other managed k8s?
|
|
#
?
May 2, 2019 04:08
|
|
|
Spring Heeled Jack posted:Openshift is just a flavor of k8s as-a-service, right? What’s so potentially bad about it vs any other managed k8s?
|
|
#
?
May 2, 2019 16:09
|
|
|
To be fair, some of those filled significant gaps in k8s at the time, and they're being phased out as upstream k8s backfilled them.
|
|
#
?
May 2, 2019 17:19
|
|
|
|
|
#
?
May 3, 2019 02:13
|
|
|
Late but thank you
|
|
#
?
May 3, 2019 09:30
|
|
|
If anyone here has some spare time to answer: What's a day in this career field like? What do you do every day, and what are some of the more difficult parts of your job?(and how do you handle them?)
|
|
#
?
May 5, 2019 09:47
|
|
|
Umbreon posted:If anyone here has some spare time to answer:
|
|
#
?
May 5, 2019 14:14
|
|
|
Umbreon posted:If anyone here has some spare time to answer: It's not markedly different than any other developer job, just a different day to day focus. It's also not supposed to be a career field, so if you work at a company that treats it as a career field, it's probably going to be pretty awful because they're playing Buzzword Bingo with no real attempt to transform company culture.
|
|
#
?
May 5, 2019 16:08
|
|
|
Bhodi posted:If you could explain what your reason for asking is, people could reply with a more tailored focus on the things you are specifically interested in. Well poo poo, I thought that was accurate enough. I've been thinking of a career field change from networking/NOC work and I've had my eye on coding related stuff, but there are so many specific jobs who all do wildly different things that I'm not sure where to start or even if I'm cut out for anything I find. I figured I'd ask around and get an idea on what people in devops do and work from there.
|
|
#
?
May 5, 2019 16:33
|
|
|
Umbreon posted:I figured I'd ask around and get an idea on what people in devops do and work from there. "Devops" is one of those awful buzzwords that means very different things to different people. For some, it basically means "remove barriers between your dev teams and your ops teams", usually this just translates to putting a single team in charge of everything from developing an app to running it in production. For others, it means the exact opposite: creating additional barriers between dev teams and ops teams. Usually this translates to creating a third "devops team" full of "devops engineers" that work somewhere between your dev team and your ops team.
|
|
#
?
May 5, 2019 17:15
|
|
|
sunaurus posted:For others, it means the exact opposite: creating additional barriers between dev teams and ops teams. Usually this translates to creating a third "devops team" full of "devops engineers" that work somewhere between your dev team and your ops team. It's even worse when a company just renames their operations team to "devops" and puts a bunch of people with somewhere between "zero" and "minimal" prior development experience in charge of writing software.
|
|
#
?
May 5, 2019 17:33
|
|
|
New Yorp New Yorp posted:It's even worse when a company just renames their operations team to "devops" and puts a bunch of people with somewhere between "zero" and "minimal" prior development experience in charge of writing software. Or even worse, do all the ceremony without changing the culture.
|
|
#
?
May 5, 2019 17:52
|
|
|
New Yorp New Yorp posted:It's even worse when a company just renames their operations team to "devops" and puts a bunch of people with somewhere between "zero" and "minimal" prior development experience in charge of writing software. Dev teams renamed to "devops teams" can also suck. I worked for my government for a while, at some point, they had decided they don't need employ as many people if they just start doing devops, so they gave their dev teams full responsibility for all ops stuff. It would have been awesome if they had actually ensured that the teams have the necessary skills/experience, but I'm guessing many didn't, because the team I got hired into most definitely had no idea what they were doing. I ended up being the first person in my team who had any kind of sysadmin experience, and the weird part is that I wasn't even hired for that experience at all (it never even came up during my interviews), it was more of a lucky coincidence.
|
|
#
?
May 5, 2019 17:53
|
|
|
Mr Shiny Pants posted:Or even worse, do all the ceremony without changing the culture. That's exactly what I'm talking about. Developers are still siloed, but now the "devops" team is responsible for making sure software that they have zero understanding of builds, deploys, and runs correctly. Meanwhile, they treat modifying a JSON or YAML file, committing it to a feature branch, and opening a PR as some sort of arcane ritual that they do, but don't understand why they're doing it or what purpose it serves. Bonus points for when developers make changes to the software that are fundamentally incompatible with the PaaS environments their software runs in, because they don't know or care what happens to their source code once they commit their changes. "But we're doing devops!!", they cry.
|
|
#
?
May 5, 2019 18:18
|
|
|
|
| # ? May 15, 2024 18:56 |
|
|
Umbreon posted:If anyone here has some spare time to answer: I'll answer your question by describing my job and providing possibly useless colour commentary! Hopefully this will give you a glimpse of what you're after, as I'll try to tailor it to your background(saw your other post). I work for an MSP("managed services provider" aka cableco), as an "AWS Developer". My team is responsible for implementing architectural best practices to drive the corporate AWS strategy, as well as evangelize cloud stuff to other teams/business units and provide them with general guidance for anything related to the cloud. We're primarily an AWS shop(like 95+%) but this could easily apply to Azure/GCP/etc in the future. I've only been on the job for a month, so I can't give the long tail of what it's like in particular . But in a previous role I was consulting to this same type of organization, so I have a decent base of reference. The majority of my time is hands-on-keyboard, writing software and templates related to cloud infrastructure management. I've also had a couple of days in workshops related to some initiatives we're trying to move forward on. Inevitably there will be a ton of troubleshooting at certain points, so putting on my systems engineer hat will be a well-practiced manoeuvre. Haven't had to do much on the evangelism tip, but I suspect that will ramp up as I become more familiar with other teams and people, and we get more automation written. Speaking of automation, that's a big focus of the team: getting tons of stuff moved away from manual touches to a state where they're largely or completely automated. This is obviously a lifetime's work, but there are some nice wins that large enterprises can afford to spend the money and time on. Doing things like automating the creation of CI/CD pipelines, the notification and remediation of "guard rails" noncompliance, internal tool building and packaging, etc can all be real force multipliers at scale. You don't have to be a software engineering genius to do this work(I'm certainly not), but solid scripting skills in something like Python or Ruby(and bash too if you have *nix in the mix) are essential. Having the understanding of a systems engineer is important, as we might be needed to troubleshoot an issue relating to on-prem systems or access, or architect a solution and take into consideration the potential complexities at a lower OSI layer or some poo poo. In general, I've found the "devops" space to be one that finally rewards generalists who are able to go deep in particular places. The real issue you're likely to encounter is, as others have mentioned, the absolute uselessness of many words orbiting the area. If you don't know what to look for, a particular position or path might seem worthwhile but actually just embody the kinds of horrors papered over by buzzword bingo. The posting for my position was remarkably terse, which in itself was promising, and I was able to telegraph a lot of info based on what they did provide. In terms of the difficult parts of the job: the #1 thing might be the constant pressure to learn and stay on top of new developments in your org's particular industry as well as tech in general. If you aren't given time on the company dime to do that learning, then they had better be paying you drat well. I heard it brought up that my company is pushing for people to be able to develop their own learning plans to move both forward and laterally in their careers/positions, but without a clear dedication to carving out the time to do that. Even with the internal support to do so, if you are somehow without the drive necessary to do that learning(whether you aren't the type of person that's naturally curious about the tech, or can't otherwise externally motivate yourself) then it's possible you'll follow the path of the dinosaur like many of the jobs people like me make obsolete! Certainly the best places reward that kind of curiosity, and the worst have a sort of slavish devotion to "learning culture" that's hollow and misunderstood. Well that's a lot of words on the topic, so I'll wrap it up here. But do feel free to come back with anything more specific, and others can probably chime in with more wisdom than I have on offer.
|
|
#
?
May 5, 2019 20:04
|
|
