|
GreenNight posted:Need to have an alerting system to alert you. When I started this job there was no alerting system, and when we did get one it was fun convincing the devs to let us know when they spun up new VM's so we could configure alerts. One of the most exciting parts of my career was when I realized infrastructure didn’t matter, and therefore it didn’t matter if devs told me they were launching new instances. To the extent we cared about any VMs at all, we have a disk space monitor that pages us if we get to 95% full on instances with the label environment:prod (or its variations, or in RGs/projects/accounts with the string “prod”). other than disk space, resource utilization really doesn’t need its own set of alarms. By all means monitor it, but those monitors should be at the cloud provider or account level, not the instance level. What matters is the service running on those machines and monitoring that service instead. That’s what APM is for first and foremost, but you can handle this however you want in your SDLC process. What do I care that a server is at 100% cpu or memory utilization if the service it provides is unaffected? By all means measure that - but you should test (and alert on!) “can service do X in Y time”.
|
#
?
Oct 3, 2023 17:20
|
|
Cat Army 
|
|
| # ? Jun 3, 2024 09:36 |
|
|
Yeah I hear you, but apparently the devs can spin up whatever they want and it's IT's problem if the instance gets a full hard drive. It was an arguement I lost.
|
|
#
?
Oct 3, 2023 17:21
|
|
|
We have alerting in place, and for all of our cloud stuff basic alerts are automatically set up. Venafi is definitely a legacy pet run by a team that doesn't seem to fully understand the service they are responsible for.
|
|
#
?
Oct 3, 2023 17:23
|
|
|
GreenNight posted:Yeah I hear you, but apparently the devs can spin up whatever they want and it's IT's problem if the instance gets a full hard drive. It was an arguement I lost. If it's in azure, between terraform enterprise and azure policy it is really hard for a team to spin up a vm with a bad config. It's all the dumb legacy on prem stuff my org struggles with.
|
|
#
?
Oct 3, 2023 17:25
|
|
|
The Fool posted:If it's in azure, between terraform enterprise and azure policy it is really hard for a team to spin up a vm with a bad config. It's all on prem stuff. The 50 PB of data we manage doesnt really do well in the cloud, from a cost perspective. Plus a lot of the VM's require big gently caress off NVIDIA cards that we stick in Nutanix hosts.
|
|
#
?
Oct 3, 2023 17:26
|
|
|
The Iron Rose posted:One of the most exciting parts of my career was when I realized infrastructure didn’t matter, and therefore it didn’t matter if devs told me they were launching new instances. To the extent we cared about any VMs at all, we have a disk space monitor that pages us if we get to 95% full on instances with the label environment:prod (or its variations, or in RGs/projects/accounts with the string “prod”). other than disk space, resource utilization really doesn’t need its own set of alarms. By all means monitor it, but those monitors should be at the cloud provider or account level, not the instance level. Yep pretty much this. I've also got some simple alarms that check disk, with some granular ones we will attach for certain services so the correct team is alerted that their thing is filling up for some reason. For the rest of the alarms I just ignore whatever other metrics the servers have going on unless it's performance impacting.
|
|
#
?
Oct 3, 2023 17:26
|
|
|
GreenNight posted:It's all on prem stuff. The 50 PB of data we manage doesnt really do well in the cloud, from a cost perspective. Plus a lot of the VM's require big gently caress off NVIDIA cards that we stick in Nutanix hosts. it's pretty funny to me that with the popularity of the cloud on prem workloads have just been left out in the cold
|
|
#
?
Oct 3, 2023 17:36
|
|
|
I worked in rural Alaska for over 5 years right out of college. One of the best decisions I ever made. I got to fly around to all of the villages and got way more responsibility handed to me than I should have with my level of experience. Plus it was like living in a different country that still spoke english and was somehow still the wild west. Todays fun: Im currently fighting a big group at my team that wants to turn off SSM patching because their over provisioned servers have CPU spikes when they patch.
|
|
#
?
Oct 3, 2023 17:38
|
|
|
The Fool posted:it's pretty funny to me that with the popularity of the cloud on prem workloads have just been left out in the cold Cloud has gotten expensive and we're dragging stuff back to on prem due to cost perspective. Our datacenter is barely a cost and it's easy as gently caress to manage.
|
|
#
?
Oct 3, 2023 17:38
|
|
|
GreenNight posted:Cloud has gotten expensive and we're dragging stuff back to on prem due to cost perspective. Our datacenter is barely a cost and it's easy as gently caress to manage. More like huge lifts and shifts with no effort to modernize or adapt to the cloud are expensive.
|
|
#
?
Oct 3, 2023 17:39
|
|
|
BaseballPCHiker posted:I worked in rural Alaska for over 5 years right out of college. One of the best decisions I ever made. I got to fly around to all of the villages and got way more responsibility handed to me than I should have with my level of experience. Plus it was like living in a different country that still spoke english and was somehow still the wild west. There is no parallel (in the united states) to what life is like in rural Alaska.
|
|
#
?
Oct 3, 2023 17:42
|
|
|
BaseballPCHiker posted:More like huge lifts and shifts with no effort to modernize or adapt to the cloud are expensive. How do you modernize when an application requires a 4 gig slice of a GPU? For each of 150 remote users? And the data accessed is in PB?
|
|
#
?
Oct 3, 2023 17:45
|
|
|
Very cool, newer hire that left "cleaned" their PC and "reset" it before their last official day, and now am being requested by other people to use that laptop for someone else internally. Yes I will totally let you use a PC I've had no look at that was reset without me having a hand in it, why do you even ask me?!
|
|
#
?
Oct 3, 2023 17:46
|
|
|
GreenNight posted:How do you modernize when an application requires a 4 gig slice of a GPU? For each of 150 remote users? And the data accessed is in PB? Oh you dont! But if you never needed elasticity or some other feature of the cloud it should have never been moved to begin with. Believe me, my org has fallen prey to the same sort of thinking. Its like the whole c-suite thought EVERYTHING in the cloud would be a no brainer slam dunk.
|
|
#
?
Oct 3, 2023 17:47
|
|
|
BaseballPCHiker posted:Oh you dont! But if you never needed elasticity or some other feature of the cloud it should have never been moved to begin with. Believe me, my org has fallen prey to the same sort of thinking. Its like the whole c-suite thought EVERYTHING in the cloud would be a no brainer slam dunk. Yeah that stuff we never moved because it wasn't cost effective. But certain workloads were moved (because THE CLOUD is the solution for everything) and then moved back because it needed to be on the same hardware as the data. There were certain workloads that were moved that made sense (.net artifacts, for example, that Jenkins utilized).
|
|
#
?
Oct 3, 2023 17:51
|
|
|
I would have loved to stay in Alaska out of high school and do "rural stuff" but it just wasn't in the cards for me. Finding a job issues and finding health care issues meant I headed down a different road and never had the option to go back. So for now I just get to be a tourist every few years.
|
|
#
?
Oct 3, 2023 17:51
|
|
|
The cloud isn’t the answer to anything or everything but I’d rather just leave the industry than work on on prem poo poo again. gently caress that noise.
|
|
|
#
?
Oct 3, 2023 17:52
|
|
|
BadOptics posted:The worst is when our HD people do this. Like, how do you expect me as the field engineer to find this location? I don't have all day to wander around an entire hospital campus. Figure it out, they're busy. Do they have to hold your hand?
|
|
#
?
Oct 3, 2023 17:55
|
|
|
The workload that GreenNight was describing seems pretty cloud-proof We've seen a pretty significant reduction in cost for most of our workloads though because being able to scale down to "almost nothing" 8 months out of the year is a huge advantage.
|
|
#
?
Oct 3, 2023 17:57
|
|
|
xzzy posted:I would have loved to stay in Alaska out of high school and do "rural stuff" but it just wasn't in the cards for me. Finding a job issues and finding health care issues meant I headed down a different road and never had the option to go back. where were you at?
|
|
#
?
Oct 3, 2023 17:58
|
|
|
The Fool posted:The workload that GreenNight was describing seems pretty cloud-proof Yeah that makes sense. I was hired to manage everything on prem. Which is fine. We're a DoD contractor and I have clearance and we have a room where classified contracts are worked on. The requirements for that room are pretty cloud-proof.
|
|
#
?
Oct 3, 2023 18:00
|
|
|
i am a moron posted:The cloud isn’t the answer to anything or everything but I’d rather just leave the industry than work on on prem poo poo again. gently caress that noise. It isn't the answer to everything but drat do I agree with you I will take whatever inconvenience over that any god drat day
|
|
#
?
Oct 3, 2023 18:00
|
|
|
Reoxygenation posted:It isn't the answer to everything but drat do I agree with you I will take whatever inconvenience over that any god drat day You don't want to install PDU's and UPS's like I did with two brand new racks yesterday?
|
|
#
?
Oct 3, 2023 18:01
|
|
|
If I didn't hate snow and ice with the firey passion of a phoenix born id probably have moved up to Alaska because I love being cold and hate the sun (I contain multitudes)
|
|
#
?
Oct 3, 2023 18:02
|
|
|
On second thought I will not take any inconvience any day.
|
|
#
?
Oct 3, 2023 18:03
|
|
|
GreenNight posted:You don't want to install PDU's and UPS's like I did with two brand new racks yesterday? You just reminded me that we have our mission critical on prem server on an UPS that nobody has tested since I started here
|
|
#
?
Oct 3, 2023 18:04
|
|
GreenNight posted:You don't want to install PDU's and UPS's like I did with two brand new racks yesterday? Don’t even know what these acronyms mean. You mean rack like your chest?
|
|
|
#
?
Oct 3, 2023 18:06
|
|
|
tokin opposition posted:hate the sun I have bad news for you about roughly 4 months of the year up there
|
|
#
?
Oct 3, 2023 18:06
|
|
|
The Fool posted:where were you at? I grew up in Anchorage, left in the mid 90's. By the time I had any IT skills it would have been a huge upheaval to go back so it just never happened.
|
|
#
?
Oct 3, 2023 18:33
|
|
|
I'd move to Alaska in a heartbeat if it wasn't for being far from family and the day/night cycle. So instead we live in the mountains in Colorado and that's pretty cool.
Internet Explorer fucked around with this message at 18:42 on Oct 3, 2023 |
|
#
?
Oct 3, 2023 18:36
|
|
|
GreenNight posted:Yeah that makes sense. I was hired to manage everything on prem. Which is fine. We're a DoD contractor and I have clearance and we have a room where classified contracts are worked on. The requirements for that room are pretty cloud-proof. So no risk of rain in that room eh? 
|
|
#
?
Oct 3, 2023 18:39
|
|
|
Internet Explorer posted:I'd move to Alaska in a heartbeat if it wasn't for being far from family and the day/night cycle. So instead we live in the mountains and Colorado and that's pretty cool. Colorado is where I finally landed. It's an Alaska that murdered all the grizzlies and shipped in humans and jeeps to replace them.
|
|
#
?
Oct 3, 2023 18:40
|
|
|
Wibla posted:So no risk of rain in that room eh? Man, the govt gave us some new regs where we have to put a garage door or a chain fence on the huge windows in that room that are 3 stories off the ground.
|
|
#
?
Oct 3, 2023 18:40
|
|
|
Anchorage isnt rural  Colorado seems nice, I landed nicely in central oregon. The high desert feels more like interior alaska than other parts of alaska.
|
|
#
?
Oct 3, 2023 18:45
|
|
|
I was in Nome and loved it. But it was time for me to move on from a career and family perspective. I had reached the ceiling of what I could do there, and my parents were getting a bit older and starting to have health issues, on top of meeting my now wife up there who grew up in Minnesota. Made it an easy decision for me to move. Some day I'd like to go back either in retirement or if/when my parents die.
|
|
#
?
Oct 3, 2023 18:45
|
|
|
Colorado has an insane housing shortfall and a lot of the front range (where most people live) is kinda gross. TABOR is loving stupid and kneecaps government programs if you ever need any (hope you don’t need help with UI while you live there!). I’d move back if I felt like being house poor and putting my kids in lovely school systems in exchange for lower property tax rates
|
|
|
#
?
Oct 3, 2023 18:49
|
|
|
The Fool posted:Anchorage isnt rural Sorry, didn't try to imply that. The dream was to graduate and find a job out in the bush but the openings just weren't there. Also my skills weren't there. 
|
|
#
?
Oct 3, 2023 18:52
|
|
|
xzzy posted:Sorry, didn't try to imply that. The dream was to graduate and find a job out in the bush but the openings just weren't there. Also my skills weren't there. Didn't think you were, just trying to make a joke. For my money, for anyone reading this that has an interest, the best way to get an IT job that will take you to actual rural locales in Alaska is to work desktop for the University system. The techs that are based out of the Fairbanks campus support all of the rural campuses as well, and while a lot of stuff is set up to be managed remotely, when they need "smart" hands they send the desktop techs.
|
|
#
?
Oct 3, 2023 19:00
|
|
|
 tag yourself, i'm emotions and packet storms
|
|
#
?
Oct 3, 2023 21:38
|
|
|
|
| # ? Jun 3, 2024 09:36 |
|
|
All my layers are busted.
|
|
#
?
Oct 3, 2023 21:39
|
|