Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
Dr. Arbitrary
Mar 15, 2006

Bleak Gremlin

Kurieg posted:

The worst I've ever done was accidentally drop a table I was trying to truncate. But that was in a test environment and we were able to trash and refresh it while we were at lunch.

The most instructive failures I was witness to, taught me the importance of version control, and that you can get away with literally anything if the CTO is using you as an excuse to expense trips to strip clubs.

The first time I saw "truncate" it was via a UI and I expected that when I clicked on it, I'd get a dialogue window to indicate how much I'd like to truncate, i.e. shorten (something) by cutting off the top or the end.

No warning, no confirmation box. All the data gone.

Adbot
ADBOT LOVES YOU

Dravs
Mar 8, 2011

You've done well, kiddo.

shortspecialbus posted:

This doesn't absolve you of risk - my story earlier was puppet code that made a bad assumption about home directories being unique that resulted in significant irreplaceable data loss.

Yeah but if I gently caress up the test environment at 5pm on a Friday I can just shrug and fix it on Monday morning. Plus my projects are all in Azure now, so no more having to do bullshit like power down tests or hardware replacement.

AlexDeGruven
Jun 29, 2007

Watch me pull my dongle out of this tiny box


Ugh. As much as I love what I do, managing physical servers can be such a pain in the rear end (even if they are hardon-inducing dual-socket 64-core Epyc2 boxes with 512gb of RAM and 2tb of nvme storage), especially when other infrastructure lets me down.

Rebooted one of those boxes and the switch corrupted the LACP config, so it never came back online. Had to rebuild the bond at both ends to get it back fully.

SyNack Sassimov
May 4, 2006

Let the robot win.
            --Captain James T. Vader


AlexDeGruven posted:

Ugh. As much as I love what I do, managing physical servers can be such a pain in the rear end (even if they are hardon-inducing dual-socket 64-core Epyc2 boxes with 512gb of RAM and 2tb of nvme storage), especially when other infrastructure lets me down.

Rebooted one of those boxes and the switch corrupted the LACP config, so it never came back online. Had to rebuild the bond at both ends to get it back fully.

I'd be happy to take it off your hands and relieve you of the stress. I'll even pay for the shipping cause I'm a real nice fella.

Kurieg
Jul 19, 2012

RIP Lutri: 5/19/20-4/2/20
:blizz::gamefreak:

Dr. Arbitrary posted:

The first time I saw "truncate" it was via a UI and I expected that when I clicked on it, I'd get a dialogue window to indicate how much I'd like to truncate, i.e. shorten (something) by cutting off the top or the end.

No warning, no confirmation box. All the data gone.

This is why I have autocommit turned off in all my instances of SQL. If it needs to be turned on because I'm doing an absurdly large batch update then a part of the script that does that batch update turns it off again afterwards.

Ghostlight
Sep 25, 2009

maybe for one second you can pause; try to step into another person's perspective, and understand that a watermelon is cursing me



We're looking at whether it's cost effective or not to increase our licensing level on a tentpole application, which will also require putting some processing and data into the cloud, but the company selling it will only help us price and scope that part of it after we commit to a three-year licensing plan, so we've had to ask around to get information on likely costs to make sure we're not over-committing for little benefit.

Posted to our Teams chat about it today are pictures received by email from the CIO of the city transit agency of two Powerpoint slides regarding their recent cost savings of AWS restructuring that were taken using a mobile phone pointed at the screen. In portrait.

Agrikk
Oct 17, 2003

Take care with that! We have not fully ascertained its function, and the ticking is accelerating.

AlexDeGruven posted:

hardon-inducing dual-socket 64-core Epyc2 boxes with 512gb of RAM and 2tb of nvme storage

How much does a server like that even cost? 128 cores in a single box? Yikes!

KillHour
Oct 28, 2007


Agrikk posted:

How much does a server like that even cost? 128 cores in a single box? Yikes!

I had something along the same lines of insanity back when I did onsite training (dual 22 core Xeon e5-2699a v4, 512GB RAM, 4TB NVMe + 12TB SAS SSD in a 1u shallow depth Supermicro pizza box) and it was like 50k IIRC.

less than three
Aug 9, 2007



Fallen Rib

door.jar posted:

Echoing everyone who has demonstrated that any good IT pro has some kind of history of loving everything up in some kind of way. It's seemingly the only way we learn. The worst people I've worked with are the ones who get burned by some stupid bullshit and either learn nothing at all or entirely the wrong lesson. My favourite was the dude who FOR YEARS insisted on typing certain commands by hand in specific ways because one time it didn't work, except it hadn't worked because of some entirely unrelated gently caress up prior to that step being run.

Yeah, I've caused a few outages. The main one where I hadn't had enough coffee and was troubleshooting something and kept issuing 'clear interface ethernet' instead of 'clear counters ethernet' which was resetting the TCP connections for 1000 people each time I did it.

Antigravitas
Dec 8, 2019

Die Rettung fuer die Landwirte:
Oh great that reminds me of that firewall appliance that just dumped the entire state table each time a rule (any rule) was changed.

Adding a few rules was a massacre. This was apparently seen as normal.

Sywert of Thieves
Nov 7, 2005

The pirate code is really more of a guideline, than actual rules.

At my first real programming job back in 2009, our lead dev setup master/slave databases but neglected to restrict access properly to the others. This inevitably resulted in a dumbass dev (hired ~2-3 months before me) doing big queries on the master instead of the slave, halting the ENTIRE network of websites that we had for about 20-30 minutes.

After this happened for the 3rd time, he got a performance review. It didn't go so well.

AlexDeGruven
Jun 29, 2007

Watch me pull my dongle out of this tiny box


Agrikk posted:

How much does a server like that even cost? 128 cores in a single box? Yikes!

The EPYC processors are insanely reasonable when it comes to the server space. I believe they were like $20-30k, which is insane considering that the builder price for 1 single 56-core Intel CPU is $25k on it's own.

If you're not going super-enterprise big-iron poo poo (like the other servers I work on at $120k/each), EPYC(2) based PowerEdge is a steal.

wolrah
May 8, 2006
what?

Antigravitas posted:

Oh great that reminds me of that firewall appliance that just dumped the entire state table each time a rule (any rule) was changed.

Adding a few rules was a massacre. This was apparently seen as normal.

I used to support a router called the Edgewater Edgemarc, and it does pretty much this. Any change that's even adjacent to the firewall results in a state table reset.

Until software version 13, which was only a few years ago, it didn't even have a way to make multiple changes at once other than hand-editing config files. Current versions at least let you queue multiple changes to apply at once.

Powered Descent
Jul 13, 2008

We haven't had that spirit here since 1969.

Agrikk posted:

How much does a server like that even cost? 128 cores in a single box? Yikes!

Setting an alarm for 15 years from now to reply "lol" to this from a cheapass 256-core chromebook.

Ham Equity
Apr 16, 2013

The first thing we do, let's kill all the cars.
Grimey Drawer

Ghostlight posted:

We're looking at whether it's cost effective or not to increase our licensing level on a tentpole application, which will also require putting some processing and data into the cloud, but the company selling it will only help us price and scope that part of it after we commit to a three-year licensing plan, so we've had to ask around to get information on likely costs to make sure we're not over-committing for little benefit.

Posted to our Teams chat about it today are pictures received by email from the CIO of the city transit agency of two Powerpoint slides regarding their recent cost savings of AWS restructuring that were taken using a mobile phone pointed at the screen. In portrait.

I feel like the only appropriate response to this is to start getting quotes for new software.

Kurieg
Jul 19, 2012

RIP Lutri: 5/19/20-4/2/20
:blizz::gamefreak:

Powered Descent posted:

Setting an alarm for 15 years from now to reply "lol" to this from a cheapass 256-core chromebook.

A core for every gigabyte of ram Chrome eats.

Thanks Ants
May 21, 2004

#essereFerrari


Ghostlight posted:

We're looking at whether it's cost effective or not to increase our licensing level on a tentpole application, which will also require putting some processing and data into the cloud, but the company selling it will only help us price and scope that part of it after we commit to a three-year licensing plan, so we've had to ask around to get information on likely costs to make sure we're not over-committing for little benefit.

Posted to our Teams chat about it today are pictures received by email from the CIO of the city transit agency of two Powerpoint slides regarding their recent cost savings of AWS restructuring that were taken using a mobile phone pointed at the screen. In portrait.

Does this company not understand that there is a cost involved in getting sales?

SyNack Sassimov
May 4, 2006

Let the robot win.
            --Captain James T. Vader


Thanatosian posted:

I feel like the only appropriate response to this is to start getting quotes for new software.

Speaking of quotes for new software - sfwarlock is there any news on what happened with our dear buddy Mitch? Did we, uh, get him?

Dirt Road Junglist
Oct 8, 2010

We will be cruel
And through our cruelty
They will know who we are

Thanks Ants posted:

Does this company not understand that there is a cost involved in getting sales?

:yaybutt:

Ghostlight
Sep 25, 2009

maybe for one second you can pause; try to step into another person's perspective, and understand that a watermelon is cursing me



Thanks Ants posted:

Does this company not understand that there is a cost involved in getting sales?
They understand the costs involved in replacing their market embedded products being higher than their licenses.

GnarlyCharlie4u
Sep 23, 2007

I have an unhealthy obsession with motorcycles.

Proof

quote:

The detectives are not able to open any email sent by another Police Department in reference to their cases or companion cases in the county for their case files. Is there a way so that the Detectives can have this encryption unlocked so they can get their work completed more efficiently?

lol :shrug:
yeah let me get right on brute forcing encryption on law enforcement's emails. I'll get back to you in a few trillion years with the progress...

Shugojin
Sep 6, 2007

THE TAIL THAT BURNS TWICE AS BRIGHT...


GnarlyCharlie4u posted:

lol :shrug:
yeah let me get right on brute forcing encryption on law enforcement's emails. I'll get back to you in a few trillion years with the progress...

I mean it's law enforcement so if the password is probably 0000 or 1234

dragonshardz
May 2, 2017

Shugojin posted:

I mean it's law enforcement so if the password is probably 0000 or 1234

Not 1312? :v:

GnarlyCharlie4u
Sep 23, 2007

I have an unhealthy obsession with motorcycles.

Proof

Shugojin posted:

I mean it's law enforcement so if the password is probably 0000 or 1234



This was the password to the evidence locker. I walked up to it and let myself in right in front of the Major and he said "wait how did you know that?"
I just said "you have to be loving kidding me right?"
They changed it to 5050 :ughh:

lament.cfg
Dec 28, 2006

we have such posts
to show you




Shugojin posted:

I mean it's law enforcement so if the password is probably 0000 or 1234

It’s definitely 1488

orange juche
Mar 14, 2012



Super Soaker Party! posted:

Speaking of quotes for new software - sfwarlock is there any news on what happened with our dear buddy Mitch? Did we, uh, get him?

I think warlock and their boss got cut out of the loop on the saga of Mitch. Mitch is probably done though in infosec role, everywhere, especially if the company is as big as it is implied to be.

Mitch might be able to get a job doing something completely unrelated though.

klosterdev
Oct 10, 2006

Na na na na na na na na Batman!
The PIN pad is full of dents, the locker is pried open, and a check-in/check-out clipboard is hanging on the wall relying on the honor system.

The pen is missing.

CommunityEdition
May 1, 2009
Didn’t need to pry it open. The code’s 0451

It’s always 0451

RFC2324
Jun 7, 2012

http 418

orange juche posted:

I think warlock and their boss got cut out of the loop on the saga of Mitch. Mitch is probably done though in infosec role, everywhere, especially if the company is as big as it is implied to be.

Mitch might be able to get a job doing something completely unrelated though.

I don't think Mitch was in infosec, I think Mitch was onsite hardware team/remote hands person

orange juche
Mar 14, 2012



RFC2324 posted:

I don't think Mitch was in infosec, I think Mitch was onsite hardware team/remote hands person

lmao so his hands shouldn't have been inside the rack unless someone specifically told him to open the rack doors anyways. It gets better and better.

For reference: I used to be the remote hands person at a colo, you're not supposed to open any rack doors unless you're specifically doing work for the customer and have a trouble ticket documenting same. If you crack a rack door open without authorization and you get found out you're hosed 6 ways to sunday.

orange juche fucked around with this message at 01:57 on Jul 15, 2020

RFC2324
Jun 7, 2012

http 418

orange juche posted:

lmao so his hands shouldn't have been inside the rack unless someone specifically told him to open the rack doors anyways. It gets better and better.

For reference: I used to be the remote hands person at a colo, you're not supposed to open any rack doors unless you're specifically doing work for the customer and have a trouble ticket documenting same. If you crack a rack door open without authorization and you get found out you're hosed 6 ways to sunday.

depends on the DC, however, it sounds like he had a ticket to do physical decommissions of some servers, found the stick in a machine and proceeded to decommission both the dongle and the licensing server that may have actually cached the license so it could be verified

so he was following directions and either got the wrong server, or failed to ask questions when a server on the list was not in the expected condition to be decommissioned in.

Arquinsiel
Jun 1, 2006

"There is no such thing as society. There are individual men and women, and there are families. And no government can do anything except through people, and people must look to themselves first."

God Bless Margaret Thatcher
God Bless England
RIP My Iron Lady

orange juche posted:

lmao so his hands shouldn't have been inside the rack unless someone specifically told him to open the rack doors anyways. It gets better and better.

For reference: I used to be the remote hands person at a colo, you're not supposed to open any rack doors unless you're specifically doing work for the customer and have a trouble ticket documenting same. If you crack a rack door open without authorization and you get found out you're hosed 6 ways to sunday.
The specific verbiage you are using here makes me think I have worked for the same colo provider.

orange juche
Mar 14, 2012



Arquinsiel posted:

The specific verbiage you are using here makes me think I have worked for the same colo provider.

I think that's standard practice, because of data security issues.

RFC2324
Jun 7, 2012

http 418

orange juche posted:

I think that's standard practice, because of data security issues.

Are we even sure he was a colo employee and not onsite for warlocks company? because it sure sounds like its the same company, in which case those colo rules don't apply

A Frosty Witch
Apr 21, 2005

I was just looking at it and I suddenly got this urge to get inside. No, not just an urge - more than that. It was my destiny to be here; in the box.
Tickets haven't come in yet because I'm in training.

I am already so much happier to be away from the piss hell that is Mississippi. This new job is so wildly different than anywhere else I've worked, I'm still half-convinced I'm going to wake up back in the television station. I did have one moment where I walked into the shop and saw a stack of Buffalo NAS units on a desk and I had to double-take.

Anyways, hopefully I can still contribute some meaningful content to this thread but I think we can consider the saga of larchesdanrew finally over.

It's been wild, y'all.

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

Hey dude I still want to hear stories from your new job even if it’s just ticket or project related. They don’t all need to be horror stories.

Neddy Seagoon
Oct 12, 2012

"Hi Everybody!"

larchesdanrew posted:

I am already so much happier to be away from the piss hell that is Mississippi. This new job is so wildly different than anywhere else I've worked, I'm still half-convinced I'm going to wake up back in the television station. I did have one moment where I walked into the shop and saw a stack of Buffalo NAS units on a desk and I had to double-take.

They know who you are!... :tinfoil:

The Fool
Oct 16, 2003


GreenNight posted:

Hey dude I still want to hear stories from your new job even if it’s just ticket or project related. They don’t all need to be horror stories.

Not empty quoting

mllaneza
Apr 28, 2007

Veteran, Bermuda Triangle Expeditionary Force, 1993-1952




orange juche posted:

I think that's standard practice, because of data security issues.

I switched from a full to a half cage at a lovely colo once and paid their hands to do the move. Next time I'm in the colo I was very surprised to find that we had half of a full-height cage. Someone else had the other half, so we had full access to each other's stuff.

Adbot
ADBOT LOVES YOU

less than three
Aug 9, 2007



Fallen Rib
One day visiting our colo, I tried to say Hi to one of their remote hands while in the little coffee room and he basically said he's not allowed to talk to me or even acknowledge where he is or what he is doing.

  • 1
  • 2
  • 3
  • 4
  • 5