|
bro just fix your busted nginx
|
#
?
Dec 15, 2022 05:02
|
|
|
|
| # ? May 18, 2024 03:44 |
|
|
MightyBigMinus posted:bro just fix your busted nginx That's a bit besides the point, because in order for traffic to get from AWS into the DC to be received by nginx, it must cross all of the intermediary pops by way of my anycast scheme anyway. Traffic can either 1) go into the ingress-nginx NodePort, and then microservice Service object or 2) The end service NodePort directly.
|
|
#
?
Dec 15, 2022 05:56
|
|
|
consul is an objectively bad design for all of the problems it purports to solve, and its failure modes are egregious. it makes docker swarm look well reasoned. if you're rolling out a new deployment of consul in 2022 then you really need to step back and ask yourself if you should be in this business.
|
|
#
?
Dec 15, 2022 09:22
|
|
|
FamDav posted:consul is an objectively bad design for all of the problems it purports to solve, and its failure modes are egregious. it makes docker swarm look well reasoned. what's good then
|
|
#
?
Dec 15, 2022 09:22
|
|
|
FamDav posted:if you're rolling out a new deployment of consul in 2022 then you really need to step back and ask yourself if you should be in this business. I checked with my Hashicorp sales rep and they told me, most certainly, to remain in this business
|
|
#
?
Dec 15, 2022 14:13
|
|
|
Vulture Culture posted:This is why HashiCorp has enterprise support, though. In a situation like this, you can have an engineer join your debugging session live and async Slack their coworkers about what to do, then slowly ask you for more difficult-to-obtain diagnostic data while not resolving the issue over a span of 24 days Don't dox me Seriously though had this happen fairly recently with datadog just about put my fist through the screen in frustration at the process
|
|
#
?
Dec 15, 2022 14:20
|
|
|
FamDav posted:consul is an objectively bad design for all of the problems it purports to solve, and its failure modes are egregious. it makes docker swarm look well reasoned.
|
|
#
?
Dec 15, 2022 15:16
|
|
|
Vulture Culture posted:Is this also that common a thread with HashiCorp products? My team has dealt with three Terraform Enterprise issues in the last few months that all stem from smooth-brained design issues failing in completely predictable ways. In the TFE case, the product doesn't actually query or reconcile branch/repo state when using VCS-driven workflows, for either Terraform project code or Sentinel policies. They try to use webhooks from the VCS provider as a make-believe delivery-guaranteed message bus, and use the streaming events (incorrectly) to create a model of the repository state on the other end. Then when something breaks, there's no way to say "okay, go get the right version", you have to root out every single failing webhook—something not designed whatsoever for these kind of delivery guarantees—and mash changes to the repository trying to get TFE to pick up the updates. Along the way it will occasionally pick weeks-old commits from the branch and try to apply them, reverting configs and destroying live infrastructure in the process. Yeah, the TFE VCS integration is completely broken and we moved to API driven deployments a while ago because of it
|
|
#
?
Dec 15, 2022 15:19
|
|
|
Works great in TFC though!
|
|
|
#
?
Dec 15, 2022 16:29
|
|
|
in kubernetes, lets say you are trying to rolling update pods that have requests to limit the pods per node, but you need to ignore that to update the pods. how do you get around that? I'm in a situation where on rolling update, some of the new pods are stuck pending because the scheduler sees the existing pods eating the resources, when obviously the final state it would replace those pods.
|
|
#
?
Dec 15, 2022 17:07
|
|
|
Vulture Culture posted:This is why HashiCorp has enterprise support, though. In a situation like this, you can have an engineer join your debugging session live and async Slack their coworkers about what to do, then slowly ask you for more difficult-to-obtain diagnostic data while not resolving the issue over a span of 24 days Does anyone have a vendor with support they consistently like? I feel like this is the same experience with basically everyone once you get beyond trivial poo poo that's in their ServiceNow knowledge base. Sometimes when you first onboard with a new vendor you get the A team for a few months and it's great, but inevitably they have a totally coincidental reorg and suddenly the extremely knowledgeable, helpful, and available people you've been working with are gone and you're dealing with a call center in Mumbai  I've had some really great interactions with AWS support, but just as many where I spend 2 hours talking to a wall because the person fundamentally misunderstands what I am asking and will not step back and listen. Just keep charging toward closing the ticket as fast as possible. If you do get an expert, though, it's loving awesome. Gitlab is the absolute worst, imo. We "have a support contract" but the outcome of every single ticket and request seems to be them adding a +1 to a 10 year old issue on their bug tracker that receives zero action or attention. I cannot think of a single thing they have fixed or helped with.
|
|
#
?
Dec 15, 2022 17:56
|
|
|
For AWS we had a weekly call with our rep + a technical resource. This has been really helpful as they can go talk to an SME about if "putting XYZ replication on the widget is on the roadmap, and if they could prioritize it please, so we can use widget in two months, instead of wasting a bunch of money developing nearly identical functionality, internally"*, also you have a neck to choke when things fall behind or start breaking, rather than an anonymous engineer behind an email address/service portal That probably only works if your service provider thinks you are a growth account, though *Actually saw features get rolled out by requesting this in meetings, not making it up
|
|
#
?
Dec 15, 2022 18:05
|
|
|
I've been having good experiences with venafi lately
|
|
#
?
Dec 15, 2022 18:17
|
|
|
Yeah we have AWS enterprise support so we do have a dedicated TAM. Maybe we just drew the short straw, though, because I hear a lot of people who are thrilled with the value and effectiveness of their TAM. Ours is... ok. He does his best but he seems stretched too thin. I feel like the vast majority of questions result in "huh I don't know about that, you should open a support case and CC me on it". Thanks bro I am really glad we are paying high 6 figures and growing a year for me to sit in chat sessions with a level 1 tech the same as I would on basic support. Or as another example I asked several times to schedule a meeting with him before Christmas to discuss something. He said he would send out an invite and then just... didn't. He has been helpful in getting calls set up with subject matter experts from time to time. But I would not give the TAM experience rave reviews. It's been no better than your account rep at any other vendor.
|
|
#
?
Dec 15, 2022 18:18
|
|
|
Aviatrix support has been really proactive and I feel bad that I'm not replying to their follow-ups because they were pretty drat helpful despite not resolving my issue because we identified the vendor that's the real problem at least (Meraki) and it doesn't even matter technically because that building flooded and we don't need to do that project anymore, lol
|
|
#
?
Dec 15, 2022 18:44
|
|
|
This is pretty vague, but does anyone know of any good books/sites/courses related to devops-adjacent best practices specific to multi-tenant white-label applications? I feel like we have issues with configuration management, feature toggles, and custom features for clients.
|
|
#
?
Dec 15, 2022 19:14
|
|
|
Trapick posted:I feel like we have issues with configuration management, feature toggles, and custom features for clients. 
|
|
#
?
Dec 15, 2022 19:17
|
|
|
Docjowles posted:Does anyone have a vendor with support they consistently like? I feel like this is the same experience with basically everyone once you get beyond trivial poo poo that's in their ServiceNow knowledge base. Sometimes when you first onboard with a new vendor you get the A team for a few months and it's great, but inevitably they have a totally coincidental reorg and suddenly the extremely knowledgeable, helpful, and available people you've been working with are gone and you're dealing with a call center in Mumbai I agree on GitLab. We've had their sales team drag for weeks on getting contracts renewed, while our install walked right past its license expiration date and into the grace period. Docjowles posted:Yeah we have AWS enterprise support so we do have a dedicated TAM. Maybe we just drew the short straw, though, because I hear a lot of people who are thrilled with the value and effectiveness of their TAM. Ours is... ok. He does his best but he seems stretched too thin. I feel like the vast majority of questions result in "huh I don't know about that, you should open a support case and CC me on it". Thanks bro I am really glad we are paying high 6 figures and growing a year for me to sit in chat sessions with a level 1 tech the same as I would on basic support. Or as another example I asked several times to schedule a meeting with him before Christmas to discuss something. He said he would send out an invite and then just... didn't. On the other stuff, it sounds like they're not super technical and don't really know what to do with the request. Sometimes this is fine, sometimes not. If you aren't getting what you need, it's probably not out of laziness, it's probably out of your TAM not understanding what you're asking for. In these cases, take the opposite approach: act like they're your boss's boss's boss, and communicate with them the same way you'd do an executive summary to the CTO asking them for something.
|
|
#
?
Dec 15, 2022 19:20
|
|
|
The quality of TAM may be heavily influenced by your total spend with the vendor. You may both be on Enterprise support, but the $100MM/year customer is going to get the velvet rope TAM treatment while the $1MM/year customer gets the intern. At any rate, the worst support is from MindTree, Azure's default 3rd party support org. They never read the ticket and just chase the close. They don't know the answers to anything. They will keep the ticket open so long that the person assigned will leave the company and you'll have to start again from scratch. And this is all exacerbated by the terrible Azure support technology that shoves everything into email but doesn't handle images or threading or collapsing quoted text, so every conversation reads like a shredded dictionary.
|
|
#
?
Dec 15, 2022 20:13
|
|
|
Vulture Culture posted:I hate to say this because I absolutely would not want someone doing this to me, but on the issue of follow-ups, you kind of have to act you're the TAM's boss, or at least like they're a consultant working on your dime (which they basically are). Sometimes stuff happens, but if they miss a committed follow-up, nudge them and if it takes more than one reminder, nudge their boss. Have calls above their level about how you're not happy with the level of engagement. As a nuclear option, your TAM is not set in stone and you can push to get a new one assigned (this obviously has career implications for your TAM, so don't do this lightly). Came here to say this. I agree with "don't do this lightly" but sometimes it's the only answer. We had a VMware TAM at the beginning of the year who did nothing but tell us to open support tickets, tell us that VMware support was good and fine and we just weren't working with them properly, and ask us when we were going to be done with our ESXi 7 upgrades. It took a couple months but we booted him off our account, and while the new guy isn't the equal of our 2019 TAM, he's at least useful.
|
|
#
?
Dec 15, 2022 20:16
|
|
|
quote:tam tam tam I'm a TAM at a smallish software company, I work only for one huge customer 100%. It's extremely hard to understand what I'm supposed actually be doing every day. I'm supposed to give them white glove service, but I'm not allowed to actually touch their environment. So I just basically get on zoom calls all day and explain what they should do, and I maintain a lab system that is a copy of their system so I can duplicate problems/configs/etc. Zorak of Michigan posted:We had a VMware TAM at the beginning of the year who did nothing but tell us to open support tickets, tell us that VMware support was good and fine and we just weren't working with them properly, and ask us when we were going to be done with our ESXi 7 upgrades. It took a couple months but we booted him off our account, and while the new guy isn't the equal of our 2019 TAM, he's at least useful. I have a customer that also has VMWare's highest level of support w/ a TAM and there's a MAJOR issue with how our platforms interact, and we have offered to get our TAM/senior devs/etc on a call, and their TAM has offered ... to open a support ticket. To speak specifically to we just weren't working with them properly, this is exactly what VMWare told us, and my manager made this a case study in what we aren't supposed to do. When a customer with a TAM calls, the TAM is supposed to kick down any barriers to getting them what they need.
|
|
#
?
Dec 15, 2022 20:24
|
|
|
I've actually had pretty good success with Microsoft support, both on the Azure side and non-Azure side. Maybe we're a big enough customer that we're getting some behind-the-scenes white glove treatment? I think it's been a while since I've done a non-Azure support case, but very often we'd have to bug our TAM to escalate the ticket out of Tier 1 hell, but from that point things have been pretty good. I've also had really fantastic support from GitHub for our GitHub Enterprise Server instance, pretty much every issue I've ever had has been "known" and if they don't have an immediate fix available, there will at least be a workaround available.
|
|
#
?
Dec 15, 2022 20:38
|
|
|
nthing the "sometimes you need to shake up your AWS account team" lever - we've had to do it, it sucks but worse is the account team not performing. You should at very least be having a dialogue with the management of your TAMs/AMs/SAs if they aren't doing what you expect of them. If you're big enough you should also seek an executive sponsor inside AWS since they can help pull strings - this probably comes when you've got a big enough EDP. edit: for good support - I've had good experience with Cloudflare (despite the, you know, nazi enabling stuff), Bridgecrew before they were borged into Palo Alto, and Databricks luminalflux fucked around with this message at 20:44 on Dec 15, 2022 |
|
#
?
Dec 15, 2022 20:41
|
|
|
So one thing I can state about a TAM is that they are supposed to be a Technical SME on the customers deployment, as opposed to calling T1 support and having to talk to someone that isn't intimately familiar with their system. A counterpart to this is a Sales "Account Manager" that is intimately familiar with the client itself from a Sales POV. Understanding that helps. When renewal comes up for a large customer already spending $1,000,000/year, it's not just like you have some random sales guy call them, you have someone with an existing relationship deal with the business side. With a TAM it's the same thing, when a technical problem comes up, you want a single point of contact that already knows the customers technical staff and environment to be the filter.
|
|
#
?
Dec 15, 2022 20:47
|
|
|
FISHMANPET posted:I've actually had pretty good success with Microsoft support, both on the Azure side and non-Azure side.
|
|
#
?
Dec 15, 2022 21:19
|
|
|
When our MSP TAM and dedicated support engineer at a job some years ago weren’t performing the way they should I had some calls with their Service Manager. Turned out the SM was all talk no action and I guess that oozed down their org. They were incredibly into ITIL, like plat sponsor for country level events. Unfortunately (for both of us) I had experience with ITIL as well. I ended up pointing out to their country director that the SM who was all high and mighty about their processes wasn’t following them at all. Or basically doing anything for us at all besides coming to our office biweekly to waste an hour of our time telling us they didn’t improve anything. We got a different SM after that. Unfortunately that was a promotion for the lacklustre performing TAM. The guy sure was trying to step up but as with any MSP they had too much on their plate. Felt sorry for him but when we’re paying for dedicated support I’m not waiting 3 days before I get an anwser on a ticket. Ended up moving almost all services away from them and kept only a few things running in a Pilot Light kinda setup. Old and new SMs left shortly after as did their principal engineer who we worked more with than our dedicated engineer.
|
|
#
?
Dec 15, 2022 21:32
|
|
|
Azure Support/TAMs at all levels have been god awful. The only thing they truly care about on every single phone call on every single ticket is how quickly can they reduce the ticket severity. Their SLAs are also meaningless. I can’t understand why anyone would willingly go with Azure. It’s so bad that we’ve had conversations with them and have told them that it’s a one of a few big reasons why we don’t increase our spend with them. That said, GrafanaLabs support and TAM teams have been great the last ~5 years. Obviously a much smaller company, so that makes sense.
|
|
#
?
Dec 16, 2022 02:40
|
|
FISHMANPET posted:I've actually had pretty good success with Microsoft support, both on the Azure side and non-Azure side. Maybe we're a big enough customer that we're getting some behind-the-scenes white glove treatment? I think it's been a while since I've done a non-Azure support case, but very often we'd have to bug our TAM to escalate the ticket out of Tier 1 hell, but from that point things have been pretty good. I have no idea how. I’m at an F1000 (maybe F500?) doing consulting and they’re closing in on (edit: i meant seven) figures a month in spend from a couple thousand a year and a half ago and we’re like 1/10th of the way through their ‘cloud transformation’ . They have the highest tier support, a designated support engineer, constantly pay for Azure Rapid Response and it’s always an absolute poo poo show dealing with Microsoft. The TAMs and everyone else have no idea what they’re talking about at any point, the designated support engineer is worthless, and absolutely no one knows what they’re talking about even if you get in touch with Product Groups. It’s been so bad they’re cancelling most the support and knocking it down a tier ASAP because unless it is blatantly obvious it’s a platform issue there is no point engaging MS on anything. i am a moron fucked around with this message at 17:50 on Dec 18, 2022 |
|
|
#
?
Dec 16, 2022 16:07
|
|
|
the cloud is 80% ZIRP finance centralization and 20% people believing the lie that this stuff is all a solved commodity and you can just 'vendor it out' and 'get support'. tam's exist like the tires they strap to the sides of barges. they're just there to absorb the marginal collisions of larger forces. zero to contribute other than being something there for you to rub up against.
|
|
#
?
Dec 18, 2022 17:41
|
|
|
lovely TAMs, sure. The good TAMs we got at AWS have been instrumental in getting us in front of PMs for service teams and helping unfuck some pretty egregious issues in SSM.
|
|
#
?
Dec 18, 2022 17:46
|
|
|
luminalflux posted:lovely TAMs, sure. The good TAMs we got at AWS have been instrumental in getting us in front of PMs for service teams and helping unfuck some pretty egregious issues in SSM.
|
|
#
?
Dec 18, 2022 18:36
|
|
|
ive appreciated some of the TAMs ive worked with as a proxy mechanism for conversing with a service team, as much as I'd like to have some engineer from EBS in my work slack I understand that it's not a reasonable ask SAs on the other hand are usually a waste of my time, especially the non-AWS ones
|
|
#
?
Dec 18, 2022 21:16
|
|
|
i am a moron posted:I have no idea how. I’m at an F1000 (maybe F500?) doing consulting and they’re closing in on (edit: i meant seven) figures a month in spend from a couple thousand a year and a half ago and we’re like 1/10th of the way through their ‘cloud transformation’ . They have the highest tier support, a designated support engineer, constantly pay for Azure Rapid Response and it’s always an absolute poo poo show dealing with Microsoft. The TAMs and everyone else have no idea what they’re talking about at any point, the designated support engineer is worthless, and absolutely no one knows what they’re talking about even if you get in touch with Product Groups. It’s been so bad they’re cancelling most the support and knocking it down a tier ASAP because unless it is blatantly obvious it’s a platform issue there is no point engaging MS on anything. Did they try escalating their issues with their TAM or whatever they call that position now?
|
|
#
?
Dec 18, 2022 21:27
|
|
|
12 rats tied together posted:SAs on the other hand are usually a waste of my time, especially the non-AWS ones I've had a couple bad SAs, or ones that didn't do much. Our current one is really good at working through a lot of the migration stuff we're dealing with going to EKS.
|
|
#
?
Dec 18, 2022 21:59
|
|
|
It's been a few years since I last had to touch K8s but by the looks of it that's going to change soonish. Any good recommendations for courses/tutorials to get back up to speed again with regards to deploying stuff? I'm not going to be repsonsible for the clusters themselves. Good resources on Prometheus are appreciated as well for some holiday reading.
|
|
#
?
Dec 19, 2022 16:04
|
|
Crosby B. Alfred posted:Did they try escalating their issues with their TAM or whatever they call that position now? Yea it’s been escalated constantly. We had one issue (entirely self inflicted) where a network team set up an expressroute to a nonprod environment in another cloud environment then handed all the IAC over to Deloittes stupid asses who proceeded to advertise bad routes to our hub and blow traffic up in a region for four hours. Asked MS for a network engineer, they promised to have one immediately, and one got assigned and started reaching out like eight hours after the incident was over. The response from the account team was like hey, yea this is embarrassing but we don’t have enough people. And that’s essentially the response we get for most every service, unless it’s like AKS and they seem to be available immediately but can’t really do that much to begin with. And some of it is entirely understandable, they aren’t there to troubleshoot these teams apps anyways. But in the times where we could’ve actually used their assistance you get the same level of customer service so I don’t think it’s anything intentional
|
|
|
#
?
Dec 19, 2022 16:15
|
|
|
LochNessMonster posted:It's been a few years since I last had to touch K8s but by the looks of it that's going to change soonish. Any good recommendations for courses/tutorials to get back up to speed again with regards to deploying stuff? I'm not going to be repsonsible for the clusters themselves. Good resources on Prometheus are appreciated as well for some holiday reading.
|
|
#
?
Dec 19, 2022 22:08
|
|
|
Vulture Culture posted:What parts of deploying stuff? As deployment more and more becomes a declaratively-orchestrated thing that behaves in standard ways, I'm increasingly seeing "deployment" just becoming "SRE". Haven’t heard the specifics yet but I’m probably going to have to help dev teams set up pipelines to deploy services/apps all the way to supporting them i to production. Helping them build observability will likely also be a part of it. Over the past years I’ve mainly been working with ECS/Fargate. Before that I worked with EKS in a fairly basic environment, no crazy Methanar-like setups. I guess I’m looking for a refresher on how to write deploy specs, check ingress options and a helm primer or something. For monitoring they’re using Prometheus/Grafana. I have a lot of experiencing setting up monitoring infra and train people on how to work with it, just haven’t come across Prom yet so I need to get up to speed quickly with that as well. Not worried too much ablut that though.
|
|
#
?
Dec 19, 2022 23:47
|
|
|
Honestly if it were me I'd learn how to do helm by installing Prometheus and grafana on a cluster as your study project. Both of those helm charts are like, platinum+++ grade charts and there's so many tutorials out there about how to install those two charts in particular, I'm sure anyone reading this thread would be successful in that task even crosseyed drunk. Installing those charts is one of the first tasks I give to interns to bring them up to speed Installing nginx is maybe half a step more difficult as you'll probably need to at least glance at some A records or gently caress with a hosts file but not super complex either
|
|
#
?
Dec 20, 2022 03:49
|
|
|
|
| # ? May 18, 2024 03:44 |
|
|
I would also suggest checking out ArgoCD for deployments to k8s. If your CI/CD experience is with something like Jenkins it feels like living in the future. You just point it at a helm chart (using a git repo) and it will go look at your cluster and figure out what needs to happen to make reality match the desired config. It can also automatically detect and repair config drift if you want it to. I’m a big fan.
|
|
#
?
Dec 20, 2022 05:09
|
|