|
Warbird posted:Probably a lot of emails and my evening being ruined. 👍 https://community.cloudflare.com/t/txt-record-breaks-wildcard-cname/150786/4 To quote RFC 1 1912 “A common mistake is thinking that a wildcard MX for a zone will apply to all hosts in the zone. A wildcard MX will apply only to names in the zone which aren’t listed in the DNS at all.” That is, if there is a wildcard MX for *.example.com , and an A record (but no MX record) for https://www.example.com , the correct response (as per RFC 1 1034) to an MX request for https://www.example.com is “no error, but no data”; this is in contrast to the possibly expected response of the MX record attached to *.example.com . With the TXT record set. The A record gets an NX response and does not actually get wildcarded to anything useful at all. Methanar fucked around with this message at 06:26 on Feb 9, 2022 |
#
?
Feb 9, 2022 03:16
|
|
|
|
| # ? May 17, 2024 01:17 |
|
|
Hi folks. I'm the sole developer for my small consultancy and I'd like to up our devops game for our next greenfield project. I was wondering if someone could suggest some reading material or otherwise help me move in the right direction. Currently we use: - git on bitbucket for source control - ubuntu on standard digitalocean droplets for host machines - docker Our apps are very-low-user-volume speciality engineering tools. We run django/postgres for the backend, vuejs SPAs for the frontend. The workflow is currently: I do all the coding locally on docker, manually deploy to a dev server for alpha testing with our clients, manually deploy to staging server, manually deploy to prod. Deployment involves git-pulling then running migrations. There is no automation. I deploy changes to prod a couple of times a year, then things usually sit for six months to a year between updates. I'm looking for ways of incrementally improving what we're doing while still being responsible for 90+% of the development plus other engineering consulting work, and want to avoid reinventing the wheel using nothing but bespoke scripts. I'd like to be able to adopt some off-the-shelf tools and be confident that any effort or money put into using them will be worth it in the long run. If I can still be using the same stack 5 years from now and never have to seriously question the technology choice then that's awesome. If adopting a technology will help me more easily onboard a couple folks this year that would also be great. Is there a current bible of tiny-rear end-basically-solo consultancy devops I should know about? Are there any tools that are becoming as ubiquitous as docker? Are puppet/ansible/terraform/vagrant/k8s/jenkins/??? things I should be looking at? Which should I start with, and which should I adopt together? Am I too small scale to need to worry about most of this poo poo?
|
|
#
?
Feb 9, 2022 19:03
|
|
|
Your first order of business should be automating your deployment pipelines. I don’t know if bitbucket has pipeline tooling, if it doesn’t you can look at circle ci, GitHub actions or azure devops pipelines.
|
|
#
?
Feb 9, 2022 19:07
|
|
|
|
|
#
?
Feb 9, 2022 19:29
|
|
|
For Django deployments, assuming you're running just a single instance in a container, run the migration as part of your container startup. i.e. if you have a docker-entrypoint.sh that you run on container startup, have it run:code:For the other parts of automation deployment, consider your pain points. Is it juggling many configurations for lots of distinct-but-similar apps? Is it the frequency with which you have to run these manual steps? Is there a risk of making errors in the manual steps? Is it setting up a consistent dev environment? If you only have a couple of these apps to manage, and you don't do it that often, you could probably automate much of it with bash scripts. When the scale goes up (# of apps, hosts, environments, frequency of deployment, etc) and you're having to manage lots of configuration, then it's time to use a tool like Ansible, Puppet, etc.
|
|
#
?
Feb 9, 2022 19:44
|
|
|
Thanks for the advice! Those are both straightforward next steps that should streamline things a lot.
|
|
#
?
Feb 10, 2022 17:13
|
|
|
What’s the Goon Take on Nomad? Seems dumb when we have 3 dozen K8s clusters to choose from, but New Dev Team insists it’s “Better” in ways they can’t articulate. They’ve never used it, or containers, before
|
|
#
?
Feb 18, 2022 02:36
|
|
|
Are there any hot tips on Jenkins integration to AD? We’ve been bickering with the AD team of our client to help us sort out what’s not playing nice, but they’re being stubborn about it. Everything seems to be fine but whenever the app tries to connect it just gets its connection reset.
|
|
#
?
Feb 18, 2022 02:42
|
|
|
Warbird posted:Are there any hot tips on Jenkins integration to AD? We’ve been bickering with the AD team of our client to help us sort out what’s not playing nice, but they’re being stubborn about it. Everything seems to be fine but whenever the app tries to connect it just gets its connection reset. For auth? Are you using LDAPs? Is the certificate valid?
|
|
#
?
Feb 18, 2022 02:49
|
|
|
No, just AD/the AD plug-in. If LDAP would be simpler then I’m fine using that, but I’ll be the first to admit near total ignorance of anything from that part of the IT world. It’s always been a black box of “use this string and don’t ask any more questions” whenever I run up against it.
|
|
#
?
Feb 18, 2022 03:13
|
|
|
Junkiebev posted:What’s the Goon Take on Nomad? Seems dumb when we have 3 dozen K8s clusters to choose from, but New Dev Team insists it’s “Better” in ways they can’t articulate. Go straight to the VP of engineering and tell them they're using a non standard tool and their project is going to double the ops budget to support their snowflake product and Shut That poo poo Right Down™. Tell the VP you're not going to support their stuff Generally I try and accommodate reasonable requests, but gently caress those guys, I hope they choke on their own ops budget
|
|
#
?
Feb 18, 2022 03:19
|
|
|
don't involve yourself with NomadHadlock posted:Go straight to the VP of engineering and tell them they're using a non standard tool and their project is going to double the ops budget to support their snowflake product and Shut That poo poo Right Down™. Tell the VP you're not going to support their stuff Junkiebev posted:They’ve never used it, or containers, before
|
|
#
?
Feb 18, 2022 04:04
|
|
|
Nomad's competitive advantage over k8s is that it's a bit easier to get started with, and that's about it. I hope it is obvious why this isn't a good reason to introduce it when you're already using k8s.
|
|
#
?
Feb 18, 2022 04:28
|
|
|
The analogy that comes to mind is that you guys are driving the diesel Toyota 4runner from top gear, and the new dev team just rolled up in a Big Wheel tricycle Shut that poo poo right the gently caress down
|
|
#
?
Feb 18, 2022 04:36
|
|
|
Hadlock posted:Shut that poo poo right the gently caress down This guy Nomads
|
|
#
?
Feb 18, 2022 04:38
|
|
|
Does the team in question have a hybrid workload or a lot of batch jobs? That's the use case where Nomad blows K8s out of the water, but I wouldn't suggest it for a fully containerized workload if you've already got K8s.
|
|
#
?
Feb 18, 2022 05:59
|
|
|
Junkiebev posted:What’s the Goon Take on Nomad? Seems dumb when we have 3 dozen K8s clusters to choose from, but New Dev Team insists it’s “Better” in ways they can’t articulate. We're using Nomad and it's fine, but as others said, if you're already running Kubernetes then it doesn't make a lot of sense. The Hashicorp stack also starts to get "contact us" pricy if you need improved access control or other "enterprise" features, I think in the ballpark of $100,000 for a Nomad or Vault license, dunno about Consul.
|
|
#
?
Feb 18, 2022 14:49
|
|
|
Also nomad isn’t like “simpler” to setup, it needs a consul cluster and potentially vault.
|
|
#
?
Feb 18, 2022 15:16
|
|
|
Yes but the tutorial on containerizing their framework they found uses Nomad, soooooooo….
|
|
#
?
Feb 20, 2022 18:15
|
|
|
Never tried Nomad but if you ever find yourself in a position where you need to choose between DC/OS and K8s just gor for the latter and never look back.
|
|
#
?
Feb 20, 2022 18:55
|
|
|
LochNessMonster posted:Never tried Nomad but if you ever find yourself in a position where you need to choose between DC/OS and K8s just gor for the latter and never look back. Seconded. Is the official install process still a shell script with several hundred megabytes of binary data baked into it and no automation? Because it used to be and it was stupid.
|
|
#
?
Feb 20, 2022 18:59
|
|
|
Erwin posted:Seconded. Is the official install process still a shell script with several hundred megabytes of binary data baked into it and no automation? Because it used to be and it was stupid. It was when I last worked with it. In the meanwhile even d2iq dumped it in favor of k8s. Also loved the 2 different custom ingress controllers that were both broken in different ways. Or their own container runtime that was compatible with several basic docker functions. It was a really bad product and I’d rather manage docker swarm clusters than that shitshow. LochNessMonster fucked around with this message at 09:14 on Feb 21, 2022 |
|
#
?
Feb 21, 2022 09:11
|
|
|
I manage a team responsible for building a Kubernetes practice within a very complicated core platform organization. People going off-script can be demoralizing, but what's worse is that it can be a giant distraction to you. Do you have your own work to do, your own impact to make, and your own outcomes to deliver that are outside of keep other teams from making sub-optimal choices? Think really hard about what the stake is that you actually have in this problem. You might look at this problem, and think something like, "if this team has time to be wasting on Nomad, then nobody is holding them accountable for making progress on real work, and doing anything of actual value to the business." There's a good chance you would be right. That also is probably not your problem to solve. Flag the issue to whatever chain of decision-makers is accountable for understanding this situation and making a call on it, then move on with your day and get back to doing kickass poo poo for people who care. You might think you're the runners of a great democracy, and you build infrastructure for everyone. The truth is that you work on patronage. Do your best work for the people who have you on speed dial. Their advocacy is way more of a force multiplier for you than a whole chain of command in the company who: a) doesn't care that you or your work exist; b) didn't ask for your opinion on something you consider yourself an expert at; and c) aren't clearly articulating their technology choices or concerns to you. There's a truism that's well-known to experienced managers, and counterintuitive to inexperienced ones: you don't over-invest in your lowest performers. You have a certain amount of hours in the week that you can invest in making somebody else successful. You can spend that time 2x-ing the productivity of a 0.25x engineer, or you can spend it 2x-ing the productivity of a 2x engineer. In the long run, which one is better for the company? The same is true for teams. So if you believe a team is prone to making poor decisions, and those decisions will undercut your success, you will actually make the situation worse if you invest your resources (on top of them investing theirs!) in making them a minimum success despite poor decision-making. At the end of the day, it's not your job to decide for the company what risks get assumed. It's your job to make sure people have the most accurate and up-to-date information available on what those risks are, so the decision-makers can make decisions. On this specific situation, here are some other things that I believe to be true:
In summation: don't sweat it too much. People will disagree with you. People will find reasons to value seemingly weird things more than they value your expertise or input or your own priorities around scaling the technical practice of a whole business. Trying to deal with those situations reactively—responding to each situation of a bad decision in progress—will ruin your ability to make progress on things that matter, for people who are relying on you. Know who your customers are. Focus on strengthening the relationships with your moneymakers. Don't get distracted trying to convert the possible micro-customer that you've already established doesn't seem to actually discuss architecture or technical decisions with anyone around them.
|
|
#
?
Feb 21, 2022 17:47
|
|
|
Erwin posted:Seconded. Is the official install process still a shell script with several hundred megabytes of binary data baked into it and no automation? Because it used to be and it was stupid. I see k8s by itself as being like the Linux kernel. It's installable by itself but you really need a distro to make it practically useful. OpenShift fills that gap. The installer is pretty easy to use, supports a bunch of different clouds, and adds a lot of value beyond the basic k8s install.
|
|
#
?
Feb 21, 2022 20:30
|
|
|
If a team wants to try $new_tech, that's often good! Experimentation can lead to discovery of potentially better solutions. But they need to acknowledge: - it's a bet. If the bet doesn't play out, do they have an exit strategy? - when is the milestone where they'll step back and re-evaluate this bet, to decide whether they commit to it or back out? - they're treading new ground. They won't be able to get support from within the company. Are they ok with that? And they should confirm that they're not adopting it for Bad Reasons: - (as someone said above) "it was the tech mentioned in the tutorial" - it's free (that can change fast, especially if they need support or go over some scale threshold) - the manager / tech lead is familiar with it (bus factor) As Vulture Culture said, I think it's best to surface these concerns with senior management and move on. Internally I'd be concerned that my team might have to add support for $new_tech if it catches on, which I'd only be ok with if I was convinced that $new_tech offered significant value.
|
|
#
?
Feb 21, 2022 21:11
|
|
|
Kelsey Hightower says that Kubernetes isn't a PaaS in itself as much as a framework for building one, which makes more sense for why it seems like there's bazillions of things that don't seem to be necessary, it looks overengineered to many users while many experienced users in the space understand why it could be necessary. The differences between installing K8S to build a PaaS and grabbing a web framework to hack around on a weekend PoC project are clearly light years apart though. To be a bit of a Devil's advocate if you're at a company that is mandating Very Bad technologies that are almost certainly not getting adopted (I'm talking to you, IBM BlueMix, and maybe for the Xooglers Google Plus) the reasons for mandates are not due to technical merits clearly but more about political and "enterprise, lol" reasons, in which case your particular organization's value system at least around the specific Very Bad technologies is not around technical criteria, so stakeholders need to understand the risks and try to plan long-term together to comply in good faith without tanking the point of creating the software in the first place.
|
|
#
?
Feb 22, 2022 05:23
|
|
|
In most (probably all?) enterprises I’ve seen, the decision to invest in any new tech was political and had less to do with being the right choice than it being something a manager or executive wanted (read: promised his buddy on the golf course). Usual excuses for buying IBM, Oracle, HP, etc garbage is “but we need support”. Which sounds nice on paper until you actually need support and it still means 3 days of downtime because they outsourced their whole support system to Ukraine which just got cutoff due to a Russian invasion. There’s no exit strat, just sit out the contract and hope its so bad the manager/exec gets flag over it and either leaves the business unit/the company or doesn’t renew the contract after 3-5 years. Most useful stuff that gets adopted quick starts as shadow IT and gains traction so fast that it’s widespread and becoming essential before management notices.
|
|
#
?
Feb 22, 2022 08:37
|
|
|
I hate how many of those empty cliches keep getting reused like they mean something. "We're sticking with Oracle because we need their database software, and using them for everything gives us one throat to choke." Have we ever been happy with their support on anything? No? What good is having this alleged one throat to choke if we can't actually put any pressure on their metaphorical windpipe?
|
|
#
?
Feb 22, 2022 15:58
|
|
|
A few months ago my org did a forced update on all managed systems purging oracle java from them because of their idiocy regarding licensing. Now everything is openjdk and feels good man. Of course we still have a few oracle databases floating around but I can appreciate not wanting to migrate production data to something new.
|
|
#
?
Feb 22, 2022 16:01
|
|
|
Nothing to make you hate Jenkins like having to work with Jenkins.
|
|
#
?
Feb 22, 2022 16:15
|
|
|
I'd like to see Jenkins go for five minutes without announcing a CVE.
|
|
#
?
Feb 22, 2022 16:18
|
|
Vulture Culture posted:Do you have your own work to do, your own impact to make, and your own outcomes to deliver that are outside of keep other teams from making sub-optimal choices? Think really hard about what the stake is that you actually have in this problem. You might look at this problem, and think something like, "if this team has time to be wasting on Nomad, then nobody is holding them accountable for making progress on real work, and doing anything of actual value to the business." There's a good chance you would be right. That also is probably not your problem to solve. Flag the issue to whatever chain of decision-makers is accountable for understanding this situation and making a call on it, then move on with your day and get back to doing kickass poo poo for people who care. So first of all, hell yes. This is very liberating once you embrace it. I've been shaping environments for people to use, issuing advice, and occasionally jumping in when people gently caress up (or get hosed up by) cloud platforms for a while now. I'd kinda forgotten what the gatekeeping IT poo poo is like expect when I interact with network teams for hybrid connectivity or someone insists their VMWare team run all the IaaS or something. I'm in the middle of a consulting engagement where I'm weaning a couple of teams off of the idea that "owning stuff" is their job versus "understanding stuff" and "getting people moving fast". I dunno if it's learned or just a personality thing but there are a lot of people in the industry who can't really conceive what you're saying and it can be exhausting. quote:You might think you're the runners of a great democracy, and you build infrastructure for everyone. The truth is that you work on patronage. Do your best work for the people who have you on speed dial. Their advocacy is way more of a force multiplier for you than a whole chain of command in the company who: a) doesn't care that you or your work exist; b) didn't ask for your opinion on something you consider yourself an expert at; and c) aren't clearly articulating their technology choices or concerns to you. I'm going to steal some form of this. Love the clear-eyed brutal honesty.
|
|
|
#
?
Feb 22, 2022 16:40
|
|
|
xzzy posted:I'd like to see Jenkins go for five minutes without announcing a CVE. I’d love for it to just goddamn work without needing umpteen plug-ins of dubious quality/support/documentation. How do you screw up your AD plug-in in such a way it can’t even connect/bind?
|
|
#
?
Feb 22, 2022 16:48
|
|
|
Zorak of Michigan posted:I hate how many of those empty cliches keep getting reused like they mean something. "We're sticking with Oracle because we need their database software, and using them for everything gives us one throat to choke." Have we ever been happy with their support on anything? No? What good is having this alleged one throat to choke if we can't actually put any pressure on their metaphorical windpipe? Yeah. Or in the same vein, somebody screaming at you to escalate <random issue> to Amazon or GitHub or something because we're a paying customer dammit and they better take us seriously. Like sure, I agree with you in the abstract, angry tech lead. Vendors should fix issues their paying customers are having. But their customer list includes like, Ford and GE and Netflix and others who are each worth literal hundreds of millions, if not billions, more than us to their bottom line. I am happy to raise a ticket which will sit at the bottom of their queue til the end of time, though if it makes you feel better.
|
|
#
?
Feb 22, 2022 21:00
|
|
|
i am a moron posted:I'm in the middle of a consulting engagement where I'm weaning a couple of teams off of the idea that "owning stuff" is their job versus "understanding stuff" and "getting people moving fast". I dunno if it's learned or just a personality thing but there are a lot of people in the industry who can't really conceive what you're saying and it can be exhausting.
|
|
#
?
Feb 22, 2022 21:41
|
|
|
There's a line with that line of thought, though. You shouldn't gatekeep any piece of technology to the point where it negatively impacts the business, and at the same time, the business does not derive a competitive advantage from every team independently investing in and solving for their own deployment pipeline simply beacuse the company employs a lot of pipeline understanders but no pipeline owners. Which services, components, facets, or whatever of your employer's technology footprint are best owned vs best collaboratively understood varies depending on way too many factors to make any strong assertions about.
|
|
#
?
Feb 22, 2022 21:48
|
|
|
imo it's more of a matter of priorities than something being always one way or the other. If someone's priority is gatekeeping to gatekeep it becomes really obvious.
|
|
|
#
?
Feb 22, 2022 22:22
|
|
|
Ok, cool. The AD issue was due to the client apparently assigning us AD domains seemingly at random meaning only like 2 people could actually show up on the AD server Jenkins was pointed at. Because they’re an Okta shop no one seems to know anything about how their stuff is set up except the third party firm that is contracted out to do their IT and cloud work; who are super not thrilled about a DevOps push and are being about as unhelpful as humanly possible. Business as usual.
|
|
#
?
Feb 22, 2022 22:38
|
|
|
12 rats tied together posted:There's a line with that line of thought, though. You shouldn't gatekeep any piece of technology to the point where it negatively impacts the business, and at the same time, the business does not derive a competitive advantage from every team independently investing in and solving for their own deployment pipeline simply beacuse the company employs a lot of pipeline understanders but no pipeline owners. I think that in an unhealthy organization, it's impossible to make strong assertions, because nobody knows what they're talking about, and in the rare event that they do, there's no trust and nobody believes them. But in a healthy org, you can rely on actual priorities and planning to guide your own decision-making. For example: if there was no standard or mandate before someone attempted to actually implement a competing technology, then having a standard or mandate wasn't a priority with buy-in from anyone whose opinion matters. However, there are other things that are priorities. When you pivot to react to someone else's technology choice, you sacrifice the intended benefit of a business priority, something that a lot of very smart people have consciously looked into, figured out the benefit of, and decided to be very important, so you can instead do a bunch of random, unplanned triage work on the business impact of something you didn't care about yesterday. There's going to be times where this stuff is really important to the organization, and an engineer makes a call like Stanislav Petrov avoiding nuclear war. But it's a lot more common that you end up with engineers who are just constantly going off on these heroic side quests, wrangling cross-functional teams of the company's most talented engineers into bullshit meetings, loving up all the messaging about business priorities, and wrecking norms around team autonomy, so they can do random work to vet and understand technology they they are literally never going to work on or with. If you want technical governance, do technical governance. Create an intentional process that works. If you find yourself jumping in front of other teams' technical decisions and making up reasons on the fly why they shouldn't move forward, you should always start from the position that you're the rear end in a top hat, doing something that you should be apologizing for and making sure doesn't happen again. Vulture Culture fucked around with this message at 23:52 on Feb 22, 2022 |
|
#
?
Feb 22, 2022 23:50
|
|
|
|
| # ? May 17, 2024 01:17 |
|
|
I don't disagree with a ton of what you're saying but, from what I can tell, the dimension upon which you're assuming a certain level of competence/scale that I do not take for granted is this part exactly:Vulture Culture posted:When you pivot to react to someone else's technology choice, you sacrifice the intended benefit of a business priority, something that a lot of very smart people have consciously looked into, figured out the benefit of, and decided to be very important, so you can instead do a bunch of random, unplanned triage work on the business impact of something you didn't care about yesterday. This presupposes a couple things that are not always true, in my experience: - every business unit making technology choices meets the requirements of supporting that technology choice for, at least, their own planned use of it - every staff member that reacts to someone else's technology choice does so by pivoting, instead of having it be explicitly their job If either of these things are untrue, you're not the rear end in a top hat for investing yourself in a technology being made without your involvement. I also don't think that every "strategic investment" in a particular technology choice is done out of malice or ignorance, I do believe that it is possible to create healthy platform ownership teams, and we are much more firmly back into "it depends" territory in the presence of either of these things.
|
|
#
?
Feb 23, 2022 01:05
|
|