Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Mo_Steel
Mar 7, 2008

Let's Clock Into The Sunset Together

Fun Shoe

BrainWeasel posted:

Hey all. I am not in IT, but I love this thread and I wanted to contribute a story about my new favorite sysadmin.

I just started a new job which primarily involves running a bunch of resource-intensive computations on various HPC clusters. Eerily similar to my experience in grad school, except there's no job queuing system and I basically get 96 cores all to myself to play around in, which is a nice change. Anyway, the other day the sysadmin comes into my team's cube farm, slightly out of breath, and says, "Hey, can you all shut down your jobs? The server room is up over 110 degrees."

Well that's not good. We all start killing processes and shut everything down like we're the President of Madagascar. The server room isn't going to cool off any time soon and there's literally nothing I can do with the clusters turned off, so I put in a ticket for metrics' sake and go home for the day.

I come in the next day not quite sure what I'll find. The sysadmin is apparently in the building, but nowhere to be seen. After about two hours, he appears and gathers up my whole team, up to and including my boss' boss, to fill us in. Obviously, the ancient HVAC system in the server room broke down. The temperature ultimately peaked at about 130 degrees -- and that's at the temperature sensor on the wall opposite the room from the actual racks, so who knows how hot the hardware itself got. Building security got an alert from the temp sensor hours earlier, but for some reason decided not to tell anybody, and the only reason the sysadmin noticed was because he happened to be walking by the server room and realized it felt super warm there. If this had happened over the weekend, the whole building could have burnt down.

The damage: nothing that was in that room will power on any more. The clusters we just finished paying off? Dead. The clusters we've been renting, for which this kind of damage is almost certainly not covered under our rental contract? Dead. The hard drives which he had spent the morning pulling from the racks and desperately testing in another machine? Dead. The backups, which, because some of the servers are dedicated to classified data, are stored in the exact same room for "security" reasons? Dead. They are going to send the drives out to a data recovery specialist and hope for the best, but the worst-case scenario is that we just lost eight years' worth of data, for both private and government contracts, worth somewhere between hundreds of millions and tens of billions of dollars in failure-to-deliver fees.

We're all standing there for a minute, wind knocked out of us, not sure what to do, when the sysadmin speaks up again.

"Does that feel bad? You all feel sick to your stomach? GOOD. When you all sit down to talk with [boss' boss' boss] next week, remember that feeling while you're explaining to him what almost happened here yesterday and why he needs to approve the funding for the actually sensible backup system I proposed two effing years ago. Now if you'll excuse me, I need to go turn the servers back on; you'll have your access back in about an hour."

Shine on, you crazy BOFH :allears:

I agree wholeheartedly with this exercise. That sinking moment where you think you might've just utterly ruined something big is like an anchor in your mind that tells you to never ever do it again, and I bet it works well for convincing people to make proactive decisions too.

Like the moment when I decided to move a set of Access queries into a macro but then didn't check to make sure they were in order before running said macro. In a production database.

Luckily I was paying attention and when the first number that came up for a record change didn't make any sense I pulled it to a halt, but I learned two valuable things immediately that I won't forget: always check your macros, and always run in test first. Even if you're confident it'll work as intended, test first. A valuable learning experience, courtesy of the pit of my stomach cramping up immediately when I realized how much worse it could've been.

Adbot
ADBOT LOVES YOU

Altimeter
Sep 10, 2003


Volmarias posted:

Yes, although if you're at the exec level you're possibly better off just not traveling to China.

Sometimes you can't avoid it - China is a big market, sometimes you have to go. Burner phones/iPads are the way to go.

Migishu
Oct 22, 2005

I'll eat your fucking eyeballs if you're not careful

Grimey Drawer

BrainWeasel posted:

Shine on, you crazy BOFH :allears:

I want to be this guys apprentice.

Seriously, I'll live on the loving streets if I have to.

Potato Salad
Oct 23, 2014
Probation
Can't post for 6 hours!

spog posted:

I've heard that some companies have a policy that when their execs travel to China, they are issued burner phones and laptops since not only do you have the usual hackers, but also sophisticated state-sponsored attacks.

Anyone come across this for real?

Confirming the presence of malware on travel machines.

Bloodborne
Sep 24, 2008

Potato Salad posted:

Confirming the presence of malware on travel machines.

The off site dudes always seem to be the most annoying when it comes to this, since they don't want to ship their poo poo back and in the land before security all have local admin. They'll straight up lie about installing poo poo until you send them the screen shot of the service running and the exe present on their machine.

captkirk
Feb 5, 2010

Migishu posted:

I want to be this guys apprentice.

Seriously, I'll live on the loving streets if I have to.

That guy could start a frigging dojo for BOFH-fu.

pr0digal
Sep 12, 2008

Alan Rickman Overdrive

Dillbag posted:

This made my butthole clench up a bit. The last time I rebooted an ISIS during production two blades refused to mount. Fortunately they were on separate chassis and we were mirrored, but I white knuckled it for two days waiting for the replacement drives to come in.

Rebooting an Avid SAN is now on my "don't ever loving do during production" list right next to "upgrade Media Composer". I feel for you if you're in broadcast, because I guess that's never not in production.

We did both in the same day! And we're a production house so we told all the editors to go home early or wait it out. It came back up quick enough, it's a 5500 so it's all integrated, if we were running a 7000 I would be hesitant to reboot the thing.

As to the upgrades we're going from 6.5.4.5 to to 7.0.4 which has been a fun process since 7.0.4 comes with its own set of bugs. Just don't go to 7.0.4.3 since that will seg fault on boot. loving Avid.

Spankmeister: I can't speak to the 7000 as I've only used that during the class but the 5500 is "made" by Avid as it's a server and storage slapped together. Avid says they play well with others but they're full of poo poo.

pr0digal fucked around with this message at 16:46 on May 11, 2015

m.hache
Dec 1, 2004


Fun Shoe
Anyone ever run into a problem where numerous (like 25) instances of Internet Explorer open up in the task manager and just drag the system down? Clearly it's an infection of some sort but I haven't been able to isolate it yet.

Once the user is gone for the day I'll be running Combofix, TSDD Killer and whatever else I have in my arsenal to see what's going on. If it doesn't catch anything I'll just flatten the machine.

Only reason why I'm hesitant is we don't have any images for these systems so it'll be a manually restore and reload of applications. Just seeing if I can avoid a few hours of mindless clicking.

The Fool
Oct 16, 2003


m.hache posted:

Anyone ever run into a problem where numerous (like 25) instances of Internet Explorer open up in the task manager and just drag the system down? Clearly it's an infection of some sort but I haven't been able to isolate it yet.

Once the user is gone for the day I'll be running Combofix, TSDD Killer and whatever else I have in my arsenal to see what's going on. If it doesn't catch anything I'll just flatten the machine.

Only reason why I'm hesitant is we don't have any images for these systems so it'll be a manually restore and reload of applications. Just seeing if I can avoid a few hours of mindless clicking.

Definitely an infection. You can also try adwcleaner and jrt. If you had a good image, you could have had it reloaded before your second scan finished.

Sirotan
Oct 17, 2006

Sirotan is a seal.


m.hache posted:

Anyone ever run into a problem where numerous (like 25) instances of Internet Explorer open up in the task manager and just drag the system down? Clearly it's an infection of some sort but I haven't been able to isolate it yet.

Once the user is gone for the day I'll be running Combofix, TSDD Killer and whatever else I have in my arsenal to see what's going on. If it doesn't catch anything I'll just flatten the machine.

Only reason why I'm hesitant is we don't have any images for these systems so it'll be a manually restore and reload of applications. Just seeing if I can avoid a few hours of mindless clicking.

If you're not in an "every user is a local admin" environment, you can probably get away with deleting and recreating their user profile. That's what I generally do instead of spending time trying to fight the infection.

m.hache
Dec 1, 2004


Fun Shoe

Sirotan posted:

If you're not in an "every user is a local admin" environment, you can probably get away with deleting and recreating their user profile. That's what I generally do instead of spending time trying to fight the infection.

I am.

At this point though if the scans don't pull anything up I gotta wipe the system. I only have 3 hours to get all of his apps up and running before I go home and it's gotta work first thing in the morning when he get's in.

BaseballPCHiker
Jan 16, 2006

Anyone have any experience with setting up multiple displays off of one computer? Had a request to get a quote for a computer that can run 8-10 displays by itself. The only thing I can think of/find to run this is something like this:
http://www.matrox.com/graphics/en/products/display_wall/mura_mpx_series/mpx44/?productTabs=0
Maybe a couple of those and then some USB to VGA/DVI adapters?

J
Jun 10, 2001

BaseballPCHiker posted:

Anyone have any experience with setting up multiple displays off of one computer? Had a request to get a quote for a computer that can run 8-10 displays by itself. The only thing I can think of/find to run this is something like this:
http://www.matrox.com/graphics/en/products/display_wall/mura_mpx_series/mpx44/?productTabs=0
Maybe a couple of those and then some USB to VGA/DVI adapters?

I'd love to hear more about this as well. Just had a request come in to run 4, possibly 5 displays....on a laptop.

RFC2324
Jun 7, 2012

http 418

J posted:

I'd love to hear more about this as well. Just had a request come in to run 4, possibly 5 displays....on a laptop.

I think anything with display port can do it, if I understand display port correctly.

deimos
Nov 30, 2006

Forget it man this bat is whack, it's got poobrain!

RFC2324 posted:

I think anything with display port can do it, if I understand display port correctly.

DisplayPort 1.2 to be specific, upto 4 1920x1080 displays at 60Hz but the video card has to be able to crank out all the pixels and the monitors need to be daisy-chainable. And you need to sacrifice a goat to the compatibility gods.

nielsm
Jun 1, 2009



Newer cards with DisplayPort output should support connecting multiple displays to one port, with some sort of splitter or chaining. Of course you might be bandwidth-limited on that so high resolutions/fast refresh might not be possible.
Apart from perhaps an ExpressCard graphics adapter that's probably your only option for laptops that doesn't involved USB-VGA things.

For desktop computers, just add more graphics cards. Even if you can't find a motherboard with enough PCI-E x16 slots you can probably find graphics cards for x4 or x1 slots. And even if you can't find that, it's actually possible to mod graphics cards with a saw. Simply saw off the excess PCI-E lanes from the card, if you can do it without damaging unrelated traces. My own success story (I was a fraction of a mm from hitting unrelated traces.)

PurpleButterfly
Nov 5, 2012

J posted:

I'd love to hear more about this as well. Just had a request come in to run 4, possibly 5 displays....on a laptop.

All the workstations in my department consist of laptops with 4 displays each. All the laptops are attached to docking stations. Two of the displays are connected to VGA parts on the docking stations, and the other two are on USB-to-VGA adapters.

Gwaihir
Dec 8, 2009
Hair Elf

BaseballPCHiker posted:

Anyone have any experience with setting up multiple displays off of one computer? Had a request to get a quote for a computer that can run 8-10 displays by itself. The only thing I can think of/find to run this is something like this:
http://www.matrox.com/graphics/en/products/display_wall/mura_mpx_series/mpx44/?productTabs=0
Maybe a couple of those and then some USB to VGA/DVI adapters?

So long as the machine has enough PCI-e slots on the motherboard, there's a few AMD cards with 6 displayport outputs (Usually called Eyefinity editions). For single slot/low performance needs there's a Quadro NVS 510 which features 4 mini-DP outputs: http://www.nvidia.com/object/nvs-510-graphics-card.html. Importantly, they can run multiple 4k displays.

Almost anything should be able to run two of those Quadro cards no problem. They don't need external PCIe power connections. Even if the only slot is a x4 or x8 slot, so long as you're not actually doing any 3d stuff, it should be OK.

guppy
Sep 21, 2004

sting like a byob

Mo_Steel posted:

I agree wholeheartedly with this exercise. That sinking moment where you think you might've just utterly ruined something big is like an anchor in your mind that tells you to never ever do it again, and I bet it works well for convincing people to make proactive decisions too.

If I'm reading that correctly, this wasn't an exercise. It was a brush with death averted only because he caught it in time.

BaseballPCHiker
Jan 16, 2006

So a videocard doesnt have to support crossfire or SLI to have two connected to one motherboard? The setup is for a professional day trader so high resolution isnt really necessary.

Wizard of the Deep
Sep 25, 2005

Another productive workday

guppy posted:

If I'm reading that correctly, this wasn't an exercise. It was a brush with death averted only because he caught it in time.

I think the "exercise" was the collective pants-making GBS threads in the conference room after the fact.

Also, I really hope someone is going to have words with the security team that decided to ignore the alert.

BaseballPCHiker posted:

So a videocard doesnt have to support crossfire or SLI to have two connected to one motherboard? The setup is for a professional day trader so high resolution isnt really necessary.

Nope. Windows will handle multiple video cards like a champ. I'd recommend staying in the same brand (NVIDIA vs ATi), just to avoid driver conflicts, but otherwise just start throwing video cards in there. USB devices will work too, but the USB 2 video adaptors were slow and a little janky, but they work in a pinch.

Wizard of the Deep fucked around with this message at 19:39 on May 11, 2015

Gwaihir
Dec 8, 2009
Hair Elf

BaseballPCHiker posted:

So a videocard doesnt have to support crossfire or SLI to have two connected to one motherboard? The setup is for a professional day trader so high resolution isnt really necessary.

Nope, I have many piece of poo poo HP/Compaq DC8300 desktops that have dual 2d quadro NVS 300s in them. No SLI needed at all, you just pop the card in and plug stuff up.

The only caveat is that dragging windows around from the monitors on one card to monitors on the second card will be laggy. Same card monitor dragging windows around is fine, it's just when you're going over the PCIe bus to the other card that you'll see a slowdown. This doesn't matter if it's just always going to be displaying the same stuff though.

I would definitely stay away from USB video anything.

(Why does the quadro NVS300 even loving exist? I have no idea. But that's a separate matter).

lampey
Mar 27, 2012

Inspector_666 posted:

I think a tie without a jacket looks less professional than a tucked in dress shirt and nice pants, and I think that "sartorial professionals" would agree with me.

I also think that asking for people to wear a suit to an interview in order to judge their ability to adhere to a dress code that isn't "Suits everyday" is stupid because anybody can put on a suit, not everybody can look presentable every day.

Pretty much I think doing things for rote reasons is stupid for everybody involved. :shobon:

There are situations that would make sense to wear a tie without a jacket. If you want to present yourself as a position of authority while still being approachable like a teacher or social worker. Otherwise a jacket without a tie is just a stopgap and should not be part of a dress code. If the situation warrants the formality a well fitting jacket comes first and then the tie. In a location where a jacket is too warm, a linen jacket is more suitable.

nothingtrend
Oct 26, 2004

So I am tasked with an encryption issue... We are migrating a client off of McAfee to bitlocker on laptops with a TPM, but what options are viable for non-tpm systems without using a flash drive, and tie in to ad for user management?

lampey
Mar 27, 2012

nothingtrend posted:

So I am tasked with an encryption issue... We are migrating a client off of McAfee to bitlocker on laptops with a TPM, but what options are viable for non-tpm systems without using a flash drive, and tie in to ad for user management?

Do you have to type in a password now with the McAfee encryption? Could you do bitlocker and just type a password in every time it boots.

hihifellow
Jun 17, 2005

seriously where the fuck did this genre come from

nothingtrend posted:

So I am tasked with an encryption issue... We are migrating a client off of McAfee to bitlocker on laptops with a TPM, but what options are viable for non-tpm systems without using a flash drive, and tie in to ad for user management?

Sophos SafeGuard relies on a synchronized AD structure for policy assignments and uses AD authentication for all modules. The only user management done outside of AD would entail manual assignment of encryption keys if that user had need of one not automatically assigned to them. It also integrates with bitlocker.

The Muffinlord
Mar 3, 2007

newbid stupie?
A ticket didn't come in, because somehow in the span of about two hours since my boss sent me a ticket, the Spiceworks install my boss and I had been testing was uninstalled and wiped from one of our auxiliary servers. I think the MSP who we're in the process of transitioning away from is to blame, but I don't know who to blame for having a server with no backups and Volume Shadow Service turned off, even if it's just a secondary domain controller serving virtually no function.

Thank god file recovery software still works on a VM. Now I get to spend Tuesday trying to unfuck this whole mess.

Proud Christian Mom
Dec 20, 2006
READING COMPREHENSION IS HARD
You should probably drop your MSP a line and say 'hey we're doing things to XXX don't worry about it'

Nintendo Kid
Aug 4, 2011

by Smythe

BaseballPCHiker posted:

So a videocard doesnt have to support crossfire or SLI to have two connected to one motherboard? The setup is for a professional day trader so high resolution isnt really necessary.

Yeah, SLI/Crossfire are specifically for using multiple cards to run a single monitor (or for some specs, multiple cards combining to run 2), splitting some of the processing load among them. When you want to run multiple monitors, you do not need the simultaneous processing for the independent monitors, so SLI/Crossfire is unneeded.

AlexDeGruven
Jun 29, 2007

Watch me pull my dongle out of this tiny box


Gwaihir posted:

(Why does the quadro NVS300 even loving exist? I have no idea. But that's a separate matter).

Because IT departments are notoriously cheap rear end bastards and OEMs saw a way to make a few extra bucks by exploiting that.

The Muffinlord
Mar 3, 2007

newbid stupie?

go3 posted:

You should probably drop your MSP a line and say 'hey we're doing things to XXX don't worry about it'

We're about to drop them a line that their services aren't needed because we have actual staff(me) for desktop IT now, and we want someone who's better suited to handle our server-side stuff.

KoRMaK
Jul 31, 2012



In zendesk, how do I tell it to stop loving sending satisfactions surveys to me as an agent that can't rate the satisfaction because I'm an agent?

Proud Christian Mom
Dec 20, 2006
READING COMPREHENSION IS HARD

The Muffinlord posted:

We're about to drop them a line that their services aren't needed because we have actual staff(me) for desktop IT now, and we want someone who's better suited to handle our server-side stuff.

Well good luck with that then, you seem to be doing a stellar job so far!

The Muffinlord
Mar 3, 2007

newbid stupie?

go3 posted:

Well good luck with that then, you seem to be doing a stellar job so far!

We're trying to really make poo poo better for everyone. Having a ticketing system that isn't their busted-rear end Autotask will make a big difference. Having servers that make some goddamn sense will help, too. What's really been the biggest thing is just having people around whose doors can be knocked on, rather than e-mailing an automated inbox that may or may not even work at all. Everyone at our MSP is capable, but they support organizations way smaller than ours, with much less dynamic demands than ours, and they're just outclassed, and having to run everything we change past them isn't going to let us get anything done. It's just a matter of acquiring all the knowledge they had about our systems from the time before we came on-board.

Wibla
Feb 16, 2011

Let's hope they documented your stuff :v:

bitterandtwisted
Sep 4, 2006




A colleague of mine visited a potential new client...

ticket resolution posted:

Travel to [company]
Initial inspection and review of current infrastructure

Spoke with Cabling guy who advised me that they moved downstairs and recently got rid of their virgin connection 

Found that server was acting as DHCP server as well as Netgear router 
Disabled DHCP on router and changed LAN settings on router to match DHCP server
Downloaded Vsphere client 
Logged in to ESXI server and started all available servers

Users still unable to log in to file server ?
Turns out company in liquidation and liquidators have removed this server.........

MJP
Jun 17, 2007

Are you looking at me Senpai?

Grimey Drawer

The Muffinlord posted:

We're trying to really make poo poo better for everyone. Having a ticketing system that isn't their busted-rear end Autotask will make a big difference. Having servers that make some goddamn sense will help, too. What's really been the biggest thing is just having people around whose doors can be knocked on, rather than e-mailing an automated inbox that may or may not even work at all. Everyone at our MSP is capable, but they support organizations way smaller than ours, with much less dynamic demands than ours, and they're just outclassed, and having to run everything we change past them isn't going to let us get anything done. It's just a matter of acquiring all the knowledge they had about our systems from the time before we came on-board.

lovely MSP?
Autotask?

You don't happen to be in the NYC area, do you?

pr0digal
Sep 12, 2008

Alan Rickman Overdrive
One of the projects (that I got roped into) in my last few days is a panic button for the front desk. The operations manager is like "well we can just call 911 like this" and proceeds to dial 911 on the front desk phone and then hangs up.

:bang:

I of course go "oh poo poo" and call dispatch back and tell them we're doing testing so I don't have cops knocking on my door.

sfwarlock
Aug 11, 2007
"Why haven't we downloaded ourselves to the cloud yet?"

In the conversation that followed, I established that the speaker

doesn't know what the cloud is or does, other than it's that thing that all the successful companies are using
doesn't know what problems it solves (or creates)
doesn't know why we might need it or what we would use it for
doesn't know how much work will be required to implement it.

but really really wants it. So much so that it's apparently one of my Director's goals to pilot a cloud system this month and have it in production exclusive of all old systems "by the beginning of the summer buying season."

("Can't we develop this Chaos Monkey thing before we online the cloud?")

Adbot
ADBOT LOVES YOU

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

Ask for a fuckton of money and say that's how much it costs.

  • Locked thread