|
Any time our DSL connection drops I tell my wife our internet is down, so I can't really get pissy about that terminology. She's not a computer person and clearly understands that I mean our connection when I say that so anyone with a lick of sense should as well. Being pedantic about the terminology there isn't a sign of being technically skilled, it's a sign that the person is a dick.
|
#
?
Dec 9, 2020 17:55
|
|
|
|
| # ? May 28, 2024 03:29 |
|
|
shortspecialbus posted:Any time our DSL connection drops I tell my wife our internet is down, so I can't really get pissy about that terminology. She's not a computer person and clearly understands that I mean our connection when I say that so anyone with a lick of sense should as well. Being pedantic about the terminology there isn't a sign of being technically skilled, it's a sign that the person is a dick. A dick or extremely socially inept, both of which are negatives to a service oriented position. I miss the days where my naturally friendly demeanor was an extreme advantage over other applicants, but it is over.
|
|
#
?
Dec 9, 2020 17:57
|
|
|
sfwarlock posted:Always scope the issue. Yeah. I can count on one hand the number of times that someone came to me with OMG HUGE SERVICE-LEVEL OUTAGE actually was a huge service-level outage. Every other time it's a subset of users--or just one user--affected. Always figure out the scope.
|
|
#
?
Dec 9, 2020 18:06
|
|
|
It had happened at that site less than a month before, so I just pointed at the actual person involved (through the glass wall of the conference room) and said "Let's say she runs in... " (The root cause was one of the staff who thought they knew more than they did "cleaning up some wiring" (read: jury-rigged daisychained unmanaged 8-port switches, because this was a temp office) and managing to create a three-switch loop. Cue the storm... ) RFC2324 posted:A dick or extremely socially inept, both of which are negatives to a service oriented position. Yeah, the really awkward bit was the next half hour of my section and then me sitting in on two more hours of the rest of the panel when I'd already decided on a hard no. sfwarlock fucked around with this message at 18:28 on Dec 9, 2020 |
|
#
?
Dec 9, 2020 18:26
|
|
|
"hello I got hacked last night do you the ISP provide a wall to stop that from happening?" no lady i'm quite sorry we really don't provide something to stop your son from downloading titties.exe and getting you cryptolockered or whatever the gently caress happened to you
|
|
#
?
Dec 9, 2020 19:15
|
|
|
Someone hasn't been reading the claims some ISPs make about how awesome their router/modem/firewall is.
|
|
#
?
Dec 9, 2020 19:28
|
|
|
Worst outage I ever caused was the day I forgot to type switch port trunk allowed vlan add. Cut my access off and a whole huge section of our network as I frantically drove 15 minutes to the site to reload the switch. Ive only made that mistake once at least...
|
|
#
?
Dec 9, 2020 19:49
|
|
|
That's a really common one, yeah. While obviously you shouldn't do that, it's poor syntax design that you can do it so easily, IMO. I once screwed up a switch stack because I was consoled into one stack, and SSHed to a switch in a different IDF to check something, and then issued the commands in the SSH window instead of the console window.
|
|
#
?
Dec 9, 2020 19:55
|
|
|
BaseballPCHiker posted:Worst outage I ever caused was the day I forgot to type switch port trunk allowed vlan add. Cut my access off and a whole huge section of our network as I frantically drove 15 minutes to the site to reload the switch. isn't there a way to confirm the changes that reverts if you don't confirm again in X minutes?
|
|
#
?
Dec 9, 2020 20:11
|
|
|
RFC2324 posted:isn't there a way to confirm the changes that reverts if you don't confirm again in X minutes? Yeah you can do reload in X and then cancel to fix that or you can use revert if you have it configured on the switch. In my case I was new, thought it’d be a quick simple thing, and did neither.
|
|
#
?
Dec 9, 2020 20:24
|
|
|
I took a subway station offline by fatfingering an IP adress earlier this year... Not my proudest moment, but at least it wasn't an underground station so there was no immediate OH gently caress reaction from anyone because the fire suppression/escape systems were offline 
|
|
#
?
Dec 9, 2020 20:33
|
|
|
Wibla posted:I took a subway station offline by fatfingering an IP adress earlier this year... this sounds like a pretty cool job, tbh
|
|
#
?
Dec 9, 2020 20:51
|
|
|
BaseballPCHiker posted:Worst outage I ever caused was the day I forgot to type switch port trunk allowed vlan add. Cut my access off and a whole huge section of our network as I frantically drove 15 minutes to the site to reload the switch. My ex-boss once "copy start run"ed a cleared startup config then reloaded when he meant to "copy run start" klosterdev fucked around with this message at 21:13 on Dec 9, 2020 |
|
#
?
Dec 9, 2020 21:06
|
|
|
I caused an outage for a customer, the techs who wired the inside of the 3PAR storage array didn't connect the power in a redundant A/B way to each disk host in the rack (they went like this) ⎯⎯⎯⎯⎯⎯⎯⎯ | A A | | B B | | A A | | B B | | A A | | B B | | A A | | B B | | A A | | B B | ⎯⎯⎯⎯⎯⎯⎯ So half the rack turned off when we switched off side A power, instantly murdering the array. Luckily I wasn't responsible for restoring it, only reporting that "hey we were doing our approved load testing on data hall B and this array on Row 18 Tile 14 had half its storage arrays turn off when we brought down side A power, probably ought to get someone to look into that".
|
|
#
?
Dec 9, 2020 21:36
|
|
|
RFC2324 posted:this sounds like a pretty cool job, tbh It's pretty cool to be able to contribute to making the mass transit system better  Sometimes leads to fun night-time activities like walking through subway tunnels! 
|
|
#
?
Dec 9, 2020 21:46
|
|
|
orange juche posted:
Why the gently caress?
|
|
#
?
Dec 9, 2020 21:59
|
|
|
Biowarfare posted:Why the gently caress? Incompetence
|
|
#
?
Dec 9, 2020 22:05
|
|
|
I took the wrong switch down once because the dude that put them in didn't tell me he labeled them like switch0 switch1 switch2 switch3 Instead of switch1 switch2 switch3 switch4
|
|
#
?
Dec 9, 2020 22:51
|
|
|
Biowarfare posted:Why the gently caress? Yeah we opened up the back door of the rack to look at the power modules when half the arrays turned off, and we loving scratched our heads at how they wired it up. It even says on the back of the door inside how to wire for redundant power, though honestly it should be obvious to anyone whose been doing physical server touching longer than a week or so. orange juche fucked around with this message at 23:51 on Dec 9, 2020 |
|
#
?
Dec 9, 2020 23:45
|
|
|
Biowarfare posted:Why the gently caress? Our DC guys took down the production database storage in the middle of the day because they "bumped a cable and it fell out, I don't know why the power wasn't redundant" Narrator: it was What actually happened, and nobody admitted to, is that he was decommissioning another piece of hardware in the rack and grabbed the wrong pair of cables. How do we know this? Because the power was redundant, and the storage guys saw both PSUs were online until exactly the same moment.
|
|
#
?
Dec 10, 2020 00:25
|
|
|
devmd01 posted:“Tell me about a time you caused an outage” Depending on how much I think the interviewer will laugh at it, I usually lead with the "I undid some hardware changes for a customer server because, after a scheduled upgrade, they asked me to 'revert back and update'" story. 
|
|
#
?
Dec 10, 2020 00:49
|
|
|
AlexDeGruven posted:Our DC guys took down the production database storage in the middle of the day because they "bumped a cable and it fell out, I don't know why the power wasn't redundant" Yeah ours was definitely a case of morons wiring it up wrong. We didnt touch anything, and just turned the whole mess over to the folks who did the root cause analysis for the incident.
|
|
#
?
Dec 10, 2020 01:01
|
|
|
Speaking of outages, I just went through the process of returning a car from carvana that got delivered yesterday and scheduled another one for delivery. In talking with the CSR, it came out that their interface was presented through Citrix. Given how many issues and delays there were, I’m gonna guess that they were having some serious Citrix farm/app/app db issues in their data center. Even better, i got called by another agent a while ago that needed to validate that the other car was scheduled for return. She then put me on hold, and about 15 minutes later the hold music dropped. No callback yet!
|
|
#
?
Dec 10, 2020 01:04
|
|
|
devmd01 posted:Speaking of outages, I just went through the process of returning a car from carvana that got delivered yesterday and scheduled another one for delivery. In talking with the CSR, it came out that their interface was presented through Citrix. Given how many issues and delays there were, I’m gonna guess that they were having some serious Citrix farm/app/app db issues in their data center. If all their stuff lives on the citrix farm then yeah Carvana is having a bad night I'd say, though Citrix Cloud says everything is fine and nothing is on fire at this moment.
|
|
#
?
Dec 10, 2020 01:08
|
|
|
A few years ago a small mountain town in the middle of nowhere got hit by a hurricane and lost internet service. The entire uplink was some garbage 200mb wireless link serving a few dozen people at most. The next day, the local tech arrived to investigate the hub and found...nothing. Just an empty concrete slab where the building used to be.
|
|
#
?
Dec 10, 2020 01:17
|
|
|
Renegret posted:A few years ago a small mountain town in the middle of nowhere got hit by a hurricane and lost internet service. The entire uplink was some garbage 200mb wireless link serving a few dozen people at most. Was the satellite dish ok? 
|
|
#
?
Dec 10, 2020 01:22
|
|
|
This little ISP built their hub out of straw
|
|
#
?
Dec 10, 2020 01:24
|
|
|
Schadenboner posted:Was the satellite dish ok? Some say that the ghost of the satellite dish still haunts that town to this day. SpoooOoOoOoOOoOooOooky
|
|
#
?
Dec 10, 2020 01:33
|
|
|
GreenNight posted:I took the wrong switch down once because the dude that put them in didn't tell me he labeled them like Most stacks start at 0?
|
|
#
?
Dec 10, 2020 01:43
|
|
|
Bob Morales posted:Most stacks start at 0? Probably? But we have a half dozen stacks is this is the only one like that. I understand in Cisco land, the first port on the first switch is gi0/1. But that's not how they're labeled or in the Visio.
|
|
#
?
Dec 10, 2020 02:29
|
|
|
Standards aside, I would certainly expect explicit documentation to be accurate.
|
|
#
?
Dec 10, 2020 03:58
|
|
|
One time I accidentally killed our ISP's backdoor fiber line into the managed switch. I did this because I accidentally jiggled the power cord on the router barely loose when I removed the redundant unused switch that it was sitting on, which was also just sitting *on top* of our 7+ foot tall server rack. I did not apologize for this because I'm not the person who decided that it was a good idea to make the loving 32x tower of power on top of our server rack, the person who told me to remove the redundant switch was.
|
|
#
?
Dec 10, 2020 04:06
|
|
|
I couldn't even tell you how many outages I've caused over the years. Once I get things in order I generally have serious uptime, but I have walked into too many places with mounds of technical debt, no documentation, no passwords, etc.
|
|
#
?
Dec 10, 2020 05:34
|
|
|
The internet is down? I hope Jen let the elders of the internet know
|
|
#
?
Dec 10, 2020 06:07
|
|
|
I bgp'd 20gbps through a 10gbps link a few times. There was a cool time where some cabling changes caused spanning tree to decide the best path to the root was often through one of the top rack switches, which wasn't supposed to happen, and that also sent again probably 20gbps over a 10gbps link.
|
|
#
?
Dec 10, 2020 06:10
|
|
|
Canuck-Errant posted:The internet is down? I hope Jen let the elders of the internet know The only unrealistic thing about that show was the incredible size of the shared office.
|
|
#
?
Dec 10, 2020 10:12
|
|
|
GreenNight posted:Probably? But we have a half dozen stacks is this is the only one like that. I understand in Cisco land, the first port on the first switch is gi0/1. But that's not how they're labeled or in the Visio. Hey, at least you had a cable going from the last switch to the first one, right? Right? Last place I worked, none of the switches had the final cable.
|
|
#
?
Dec 10, 2020 14:08
|
|
|
klosterdev posted:This little ISP built their hub out of straw ~12 years ago I was working IT at a lovely local telco. One of the telco guys and I had a running joke-narrative about how we were going to retire to Dominica, set up a satellite ISP, and train monkeys to lay fiber-optic cable around the island. 
|
|
#
?
Dec 10, 2020 14:33
|
|
|
Bob Morales posted:Hey, at least you had a cable going from the last switch to the first one, right? Right? Yeah the stacks were stacked appropriately. We even have Meraki stacks now.
|
|
#
?
Dec 10, 2020 14:36
|
|
|
|
| # ? May 28, 2024 03:29 |
|
|
I once managed to take down 50 security camera systems at once at a former employer. I was taking over management of that customer's systems and while auditing to see what my predecessor had done/not done, the CTO came to me and said I needed to push a patch for some POS integration right now for some stores. I knew nothing about the patch, but the CTO assured me I just needed to put it in the mass patch management tool they had cobbled together. So I did. While that patch was in the process of going out, I happened to mention to the CTO that there were six months of Windows updates in the WSUS server that had never been sent to the DVRs at the stores. At which point he turned white as a sheet and told me to kill the update. Turns out that the patch required certain updates in Windows to function. Which hadn't been done. I managed to kill it before taking everything off line but 50 stores had their security camera systems go offline.
|
|
#
?
Dec 10, 2020 16:08
|
|