Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Inspector_666
Oct 7, 2003

benny with the good hair

GreenNight posted:

What, Google Authenticate was too hard?

They already offered 2FA that worked with Google Authenticate. It just wasn't well advertised.

Also people claimed to get hacked even with it enabled :iiam:

Adbot
ADBOT LOVES YOU

sloshmonger
Mar 21, 2013

nexxai posted:

So I got loving tired of losing sleep (for real) thanks to the various ransomware out there and decided to try and do something about it.

If you aren't using the File Server Resource Manager role on your Windows Servers (2008+), you should get on it immediately. It is 100% free and can monitor for filenames and cut off access to users accessing network shares if it detects too many files matching a given set of filters. Which is great because a lot of ransomware uses a specific filename pattern or file extension.

I built a site that tries to log as many of these different file types as we know about and are always ready to take submissions for ones we don't using the form at the bottom of the page. I only spent 1 day on the site so it may not be perfect but I think it gets the job done. If you have any suggestions, I would be more than happy to take them - I want to make this a global resource and the more accurate information we have, the better.

Link: https://fsrm.experiant.ca/

Let me know what you think.

Just wanted to say I put that in place a few days ago, and yesterday discovered a folder on my network from before I started with locky files in it. Tried to interact with one of the files, and suddenly I couldn't do poo poo. :eng99:

This is going everywhere.

nexxai
Jul 17, 2002

quack quack bjork
Fun Shoe

sloshmonger posted:

Just wanted to say I put that in place a few days ago, and yesterday discovered a folder on my network from before I started with locky files in it. Tried to interact with one of the files, and suddenly I couldn't do poo poo. :eng99:

This is going everywhere.

Glad to hear it's useful for you. Please spread it around to admin friends if you can. The more people that are protected, the less these assholes can make from their schemes.

Judge Schnoopy
Nov 2, 2005

dont even TRY it, pal

sloshmonger posted:

Just wanted to say I put that in place a few days ago, and yesterday discovered a folder on my network from before I started with locky files in it. Tried to interact with one of the files, and suddenly I couldn't do poo poo. :eng99:

This is going everywhere.

To expand on this, I'm looking to deploy this on our file servers and the question came up of "What happens during a detection". Is the user's AD account locked? Is that account just locked out of making changes to the server? In what way is this done?

Is the whole file server locked down?

Mo_Steel
Mar 7, 2008

Let's Clock Into The Sunset Together

Fun Shoe

Judge Schnoopy posted:

To expand on this, I'm looking to deploy this on our file servers and the question came up of "What happens during a detection". Is the user's AD account locked? Is that account just locked out of making changes to the server? In what way is this done?

Is the whole file server locked down?

The link from the site to the PS script has some more details:

quote:

How it Works

If the user writes a malicious file (as contained in the file group) to a network share, FSRM will run the deployed script which will add a Deny permission for that user against every share.

This has been tested fairly thoroughly, and I find that at most ransomware ends up encrypting one directory before the user is blocked.

The script has now been modified to pull the list of extensions from a JSON API. Credit to https://fsrm.experiant.ca/ for this list. Make sure you review the list (https://fsrm.experiant.ca/api/v1/get) before deploying, in case any false positives are listed (e.g. I have seen CAD software legitimately use *.encrypted before). When this list is updated, review it and simply run the script again to redeploy.

[...]

An event will be logged by FSRM to the Event Viewer (Source = SRMSVC, Event ID = 8215), showing who tried to write a malicious file and where they tried to write it. Use your monitoring system of choice to raise alarms, tickets, etc for this event and respond accordingly.

It detects the user writing bad files and cuts them off. Pretty slick, I'd bet it probably saves a ton of time in terms of rolling backups if you only have to do it for a couple of directories and a single user machine vs. an entire file server. Now I want to combine it with the PS script posted earlier to send voice messages on Crypto detection to the offending user. :unsmigghh:

Mo_Steel fucked around with this message at 20:14 on Jun 10, 2016

nexxai
Jul 17, 2002

quack quack bjork
Fun Shoe

Mo_Steel posted:

The link from the site to the PS script has some more details:


It detects the user writing bad files and cuts them off. Pretty slick, I'd bet it probably saves a ton of time in terms of rolling backups if you only have to do it for a couple of directories and a single user machine vs. an entire file server.

To clarify:

- Assuming you set an Active file screen, FSRM will outright prevent any user from creating or modifying any file that matches a given file screen (*.xyz) in its base configuration
- You can set up email notifications to go to the administrator (and optionally the user) explaining that a file screen was matched and that a file was not created or modified because of it
- If you use m-dwyer's script, it will also block that person from using any share on that server, preventing them from doing any further damage since many times ransomware is delivered through an exploit kit which may deliver other bad poo poo as well

pixaal
Jan 8, 2004

All ice cream is now for all beings, no matter how many legs.


It mentions that it makes an event in the event log. What is the Event ID?

SeaborneClink
Aug 27, 2010

MAWP... MAWP!

pixaal posted:

It mentions that it makes an event in the event log. What is the Event ID?

8125, I think he added that in an edit.

Relyssa
Jul 29, 2012



larchesdanrew posted:

Is that my fate?

Will I wake up and realize that to be successful I have to become that which I hate?

Didn't someone already suggest that CE is you from the future? This is sounding more and more plausible.

sloshmonger
Mar 21, 2013

Judge Schnoopy posted:

To expand on this, I'm looking to deploy this on our file servers and the question came up of "What happens during a detection". Is the user's AD account locked? Is that account just locked out of making changes to the server? In what way is this done?

Is the whole file server locked down?

It added an deny entry for the Share permissions (not ntfs permissions) for the account that did the oopsie. Recovering was going through 20 shares, finding the deny entry, hitting Remove. About 3 minutes work because I'm slow.

From the user side, I got a "you don't have permission to this share" which should be familiar. I quickly checked one of my other shared folders and got that same message, and deduced what was happening.

MisterZimbu
Mar 13, 2006

Kaethela posted:

Didn't someone already suggest that CE is you from the future? This is sounding more and more plausible.

"I have come to you from the future with a warning."

"Why? What happens in the future?"

"...It doesn't concern you."

Proteus Jones
Feb 28, 2013



MisterZimbu posted:

"I have come to you from the future with a warning."

"Why? What happens in the future?"

"...It doesn't concern you."

:aaaaa:

BOOTY-ADE
Aug 30, 2006

BIG KOOL TELLIN' Y'ALL TO KEEP IT TIGHT

nexxai posted:

Glad to hear it's useful for you. Please spread it around to admin friends if you can. The more people that are protected, the less these assholes can make from their schemes.

I passed this to our server admins to look over, because gently caress ransomware right in its dirty rear end in a top hat. At least with this it could help track down who potentially got the infection first, my biggest peeve is people that don't admit to loving up and having concrete evidence would be stellar for problem clients.

GreenBuckanneer
Sep 15, 2007

I wish my company would just come out and drop 2003/XP already.

We've already "unoffically" dropped support of some of our big features either because certain per-requisite software wont run unless they're using an updated version of that software which wont run on xp/2003, and we officially support certain windows programs which you can't install officially on XP/Vista. On top of that, they've said down the pipe we'll stop supporting it, but instead grandfather those people in.

gently caress what are you doing, just loving throw that poo poo out the window. I wish they would just drop XP/Vista support entirely, so we can get the last vestiges of these fuckers to upgrade their poo poo already.

ConfusedUs
Feb 24, 2004

Bees?
You want fucking bees?
Here you go!
ROLL INITIATIVE!!





GreenBuckanneer posted:

I wish my company would just come out and drop 2003/XP already.

We've already "unoffically" dropped support of some of our big features either because certain per-requisite software wont run unless they're using an updated version of that software which wont run on xp/2003, and we officially support certain windows programs which you can't install officially on XP/Vista. On top of that, they've said down the pipe we'll stop supporting it, but instead grandfather those people in.

gently caress what are you doing, just loving throw that poo poo out the window. I wish they would just drop XP/Vista support entirely, so we can get the last vestiges of these fuckers to upgrade their poo poo already.

Sadly, no company in their right mind is going to just tell a significant portion of their userbase to gently caress off. I also wish my company would do something of that nature, but I totally understand why they do not.

A pretty intelligent way of doing it is to grandfather existing customers. They can keep using your poo poo, usually an older version, until they go away. New versions/features are out the window, and new customers can't buy/install/whatever on those oldass platforms. I'm pushing pretty heavily for this approach, if only so we can update our products to use features that simply do not work on XP.

GreenBuckanneer
Sep 15, 2007

ConfusedUs posted:

Sadly, no company in their right mind is going to just tell a significant portion of their userbase to gently caress off. I also wish my company would do something of that nature, but I totally understand why they do not.

A pretty intelligent way of doing it is to grandfather existing customers. They can keep using your poo poo, usually an older version, until they go away. New versions/features are out the window, and new customers can't buy/install/whatever on those oldass platforms. I'm pushing pretty heavily for this approach, if only so we can update our products to use features that simply do not work on XP.

I don't really mean to tell them to fuckoff. That's wishing pigs could fly. I just want them to stop being surprised when they call support and finding our their poo poo doesn't work because xyz :(

On a side note: I really wish people would stop calling me telling me our program doesn't work, only for me to remote in and tell them the reason it doesn't work is because their OS is having problems. They then say "well your program doesn't work for me, so I should look into someone else"

I'm sorry that you're blaming your doctor that your lungs aren't working, because you notice you're getting emphysema, so instead of smoking Camels you move to Marlboro.

GreenBuckanneer fucked around with this message at 06:50 on Jun 11, 2016

Dr. Arbitrary
Mar 15, 2006

Bleak Gremlin

flosofl posted:

Hahaha. Looks like you're swearing.

What's going on here?

ConfusedUs
Feb 24, 2004

Bees?
You want fucking bees?
Here you go!
ROLL INITIATIVE!!





GreenBuckanneer posted:

I don't really mean to tell them to fuckoff. That's wishing pigs could fly. I just want them to stop being surprised when they call support and finding our their poo poo doesn't work because xyz :(

On a side note: I really wish people would stop calling me telling me our program doesn't work, only for me to remote in and tell them the reason it doesn't work is because their OS is having problems. They then say "well your program doesn't work for me, so I should look into someone else"

I'm sorry that you're blaming your doctor that your lungs aren't working, because you notice you're getting emphysema, so instead of smoking Camels you move to Marlboro.

Some people are their own worst enemies.

One time a company I was working for was sued because he couldn't restore his backup. Dude had deleted all his existing backups, because he didn't have room for new ones. Then his computer died while the next backup was in progress. So there were no complete backups, just a partial that contained only a tiny fragment of his data.

Totally his fault for cheaping out on storage to store his new backups. We spent like 6 months in court over it.

GreenBuckanneer
Sep 15, 2007

ConfusedUs posted:

Some people are their own worst enemies.

One time a company I was working for was sued because he couldn't restore his backup. Dude had deleted all his existing backups, because he didn't have room for new ones. Then his computer died while the next backup was in progress. So there were no complete backups, just a partial that contained only a tiny fragment of his data.

Totally his fault for cheaping out on storage to store his new backups. We spent like 6 months in court over it.

I just dealt with someone using our server software that made me so frustrated. Customer's old full failed to upload (disk finished, but server got rebooted during the upload, so that failed) and subsequent full he kicked off (because of the failed upload) failed because he ran out of disk storage.

The L1 agent told him "oh just start a full, that's what you need to do", and put down not to escalate it to L2. Customer hangs up on him, calls back, and L1 who was sitting with a manager gets told by the manager to escalate it to L2 immediately (without updating notes which would be bad via company policy otherwise). It goes to me, and I spend 40 minutes reviewing the case notes, the computer, the error report that our software made, the logs, and what causes the error report in the first place. After all of that, I realized that the reason for the failure was because his server kept rebooting (though once or twice was because of a power outage, but there was 6 other occurrences during the period of him backing up that the server rebooted, which I found out via powershell). What was the answer? loving DO A NEW FULL.

The customer then proceeds to try to rip into me saying our product is defective. I explain his server rebooted several times during the upload, and as such we couldn't resume the cloud upload so he has to do it all over again (which takes him weeks on his lovely internet). He then goes "no, it failed because i ran out of local storage, it's your fault it can't just resume where it left off". To which I tell him "no, your first full to the cloud failed because you rebooted. your second full to the disk failed because you ran out of local storage space, along with another incremental you tried doing. This is the error you see online, and ultimately the reason for the failure in the first place is because you rebooted the server. Stop doing that". His response was "I guess I'll do a full...i have no choice...." :commissar:

On a similar note: we had a guy who was trying to do a restore with our imaging software, only to flip the gently caress out when he needs to use other software because his system is weird, then tries blaming it on us when for some reason when he restores his keyboard and mouse isn't working. It's now as high up as it can go. I'm following that case because if it was me I would have told this guy to gently caress off, unequivocally. But management is like "buh he's a lawyer! he might sue us!!" gently caress that idiot. #sand.

SubjectVerbObject
Jul 27, 2009

Dr. Arbitrary posted:

What's going on here?



I think they are just bashing each other.

Judge Schnoopy
Nov 2, 2005

dont even TRY it, pal

BOOTY-ADE posted:

I passed this to our server admins to look over, because gently caress ransomware right in its dirty rear end in a top hat. At least with this it could help track down who potentially got the infection first, my biggest peeve is people that don't admit to loving up and having concrete evidence would be stellar for problem clients.

Look at the owner of any ransom file to find the culprit account. The problem isn't with identification, it's reaction time. When we got hit it took me 3 minutes from the first call to shutting down the machine, but that was 45 minutes after the attack started and everything was gone.

Dunno-Lars
Apr 7, 2011
:norway:

:iiam:



Dr. Arbitrary posted:

What's going on here?



The serious answer is that claiming passwords just show up as stars is a common scam, at least in online games. They make the target believe it is true and have them write their password, then you copy the thing back and change it to stars to keep them satisfied while you change all their account info, or steal their stuff.

It was common enough in Runescape that they actually added a filter to stop you from typing your password. But instead of starring it, the message just won't go through and you get a message. At least back when I played it.

Entropic
Feb 21, 2007

patriarchy sucks

Dunno-Lars posted:

The serious answer is that claiming passwords just show up as stars is a common scam, at least in online games. They make the target believe it is true and have them write their password, then you copy the thing back and change it to stars to keep them satisfied while you change all their account info, or steal their stuff.

It was common enough in Runescape that they actually added a filter to stop you from typing your password. But instead of starring it, the message just won't go through and you get a message. At least back when I played it.

That joke dates back to IRC, if not earlier.

BOOTY-ADE
Aug 30, 2006

BIG KOOL TELLIN' Y'ALL TO KEEP IT TIGHT

GreenBuckanneer posted:

I wish my company would just come out and drop 2003/XP already.

We've already "unoffically" dropped support of some of our big features either because certain per-requisite software wont run unless they're using an updated version of that software which wont run on xp/2003, and we officially support certain windows programs which you can't install officially on XP/Vista. On top of that, they've said down the pipe we'll stop supporting it, but instead grandfather those people in.

gently caress what are you doing, just loving throw that poo poo out the window. I wish they would just drop XP/Vista support entirely, so we can get the last vestiges of these fuckers to upgrade their poo poo already.

We already do this and so far, only had a couple clients (new ones) that had maybe 2-3 XP machines and one had a 2003 server they were using for backups. My only gripe was the client with the 2003 server, primarily because we got an alert last week on that box about a failed hard drive and have been pushing them to virtualize the drat thing already. Even told them that yes, the old Dell PowerEdge they're using is out of warranty and Dell doesn't even sell or support it any longer, and Server 2003 has been end of life for months with zero MS support. If it dies, they're pretty much screwed and it'll have to be rebuilt on new hardware anyways, and it's in our contract that we will not support expired/end of life hardware if the vendor no longer makes/supports it.

The client company's VP, of course, asks how much a new drive costs to replace the one that's failing :doh: our response after dealing with this same kind of garbage for months was giving 2 choices - either virtualize the server ASAP, or buy the drive on their own and replace it. Haven't heard back yet on their choice.

AlphaKretin
Dec 25, 2014

A vase to face encounter.

...Vase to meet you?

...

GARVASE DAY!

Entropic posted:

That joke dates back to IRC, if not earlier.

Yup. I don't know if http://bash.org/?244321is the origin but it's certainly from where it was popularised.

Lightning Jim
Nov 18, 2006

Just a mad weather-ologist :science:

BOOTY-ADE posted:

We already do this and so far, only had a couple clients (new ones) that had maybe 2-3 XP machines and one had a 2003 server they were using for backups. My only gripe was the client with the 2003 server, primarily because we got an alert last week on that box about a failed hard drive and have been pushing them to virtualize the drat thing already. Even told them that yes, the old Dell PowerEdge they're using is out of warranty and Dell doesn't even sell or support it any longer, and Server 2003 has been end of life for months with zero MS support. If it dies, they're pretty much screwed and it'll have to be rebuilt on new hardware anyways, and it's in our contract that we will not support expired/end of life hardware if the vendor no longer makes/supports it.

The client company's VP, of course, asks how much a new drive costs to replace the one that's failing :doh: our response after dealing with this same kind of garbage for months was giving 2 choices - either virtualize the server ASAP, or buy the drive on their own and replace it. Haven't heard back yet on their choice.

$100-$300 HDD vs $1000s for a new server.
That'll be as much as though that'll put into that.
How old is that server, anyways? 9th gen?

Dr. Arbitrary
Mar 15, 2006

Bleak Gremlin

Dunno-Lars posted:

The serious answer is that claiming passwords just show up as stars is a common scam

I guess my joke was a little too subtle, or just not very funny.

If you look at the post I took a screenshot of, the passwords are clearly in plaintext, but my screenshot has them obfuscated.

Methanar
Sep 26, 2013

by the sex ghost

Dr. Arbitrary posted:

I guess my joke was a little too subtle, or just not very funny.

If you look at the post I took a screenshot of, the passwords are clearly in plaintext, but my screenshot has them obfuscated.

I liked it once I understood it.

Proteus Jones
Feb 28, 2013



Dr. Arbitrary posted:

I guess my joke was a little too subtle, or just not very funny.

If you look at the post I took a screenshot of, the passwords are clearly in plaintext, but my screenshot has them obfuscated.

And thanks for that. It's what I had been aiming for if I had been smart enough to do it myself.

And the Hunter2 schtick will never not be funny.

Collateral Damage
Jun 13, 2009

Inspector_666 posted:

They already offered 2FA that worked with Google Authenticate. It just wasn't well advertised.

Also people claimed to get hacked even with it enabled :iiam:
:10bux: says it's because those people had the same password on their google account.

Samizdata
May 14, 2007

flosofl posted:

And thanks for that. It's what I had been aiming for if I had been smart enough to do it myself.

And the Hunter2 schtick will never not be funny.

Hunter12 is SO much more secure!

A Pinball Wizard
Mar 23, 2005

I know every trick, no freak's gonna beat my hands

College Slice
I'm working right now with a client running XP and Win 2000 to get him from version 8 to (current, win 0 compatible) version 15 of our software. It is taking multiple weeks and involves a temporarily borrowed 2008 server to bridge the gap between 2000 and 2012 and the ticket has already been open for 3 weeks and my boss is sending out departmental emails reminding everyone that calls open more than two weeks need to get TOP PRIORITY and :suicide:

Also a bunch of our customers have been unexpectedly updated to Windows 10 because our bread and butter is little mom and pop stores who run win 7pro with no domain. CNET must have gotten a huge bump in traffic just from us linking the rollback instructions.

A Pinball Wizard fucked around with this message at 03:44 on Jun 12, 2016

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

What, do you work for Quicken or something? Good luck with your fires.

MJP
Jun 17, 2007

Are you looking at me Senpai?

Grimey Drawer
A clusterfuck came in...

We have the old file server and a future file server. They both have access to a Dell SAN array. Apparently this is a bad thing, unless they're clustered.

Both servers have iSCSI sessions to the same volume on the SAN.

I tried disconnecting the iSCSI session on Old Server to make sure New Server could still read/write. Yep, it can, no problem.

I disconnected the session on New Server and reconnected on Old Server.

The drive showed back up but is corrupted and unreadable :smith:

Called Dell support, they basically said "yeah you can't do that, restore from backups" and apparently our Friday PM backups are bad.

At least Thursday's look ready to go, but chkdsk is running. It does see that the drive is an NTFS volume, and it's doing stuff, but it just went to "Inserting an index entry into index $0 of file 25" for many many lines, repeating fairly constantly so far. :ohdear:

So yeah, looks like I broke NTFS. I wonder how long it'll take to restore and when I should say gently caress it with chkdsk. My initial estimates based on around 50ish entries per second being scanned was less than an hour, which I liked better than Windows' estimate of 253 hours and increasing. +

poo poo OK it's at the point of analyzing security descriptors and recreating them, so at least things are happening. I'd rather not wipe the volume out and restore until I know poo poo's permanently hosed.

Wait now it's past that, yay, OK, things are happening, I should stop e/ning this post. I didn't even have to edit this, it's happening in real time.

Is it obvious that this is babby's first time being actually in charge of a SAN?

Thanks Ants
May 21, 2004

#essereFerrari


MJP posted:

Both servers have iSCSI sessions to the same volume on the SAN.

:stare:

MJP
Jun 17, 2007

Are you looking at me Senpai?

Grimey Drawer

This was done by me, with no knowledge of the consequences, about a month or two ago.

I now know better. Don't futz with a SAN, call the vendor if you're not sure, even if it looks intuitive and carries no obvious consequences.

I have a tape restore going to another location as a fallback measure. The company culture is tolerant of mistakes, so long as you learn from them, and my boss was OK with me making this test today. Nobody works on the weekends so we at least have 21 hours to try to mitigate damage.

Edit: gently caress Arcserve, gently caress Arcserve d2d/dedup backups in particular. They're running a restore at 19 mb/min. The tapes are going at 2251 mb/min.

Chkdsk is on stage 4, looking for bad clusters in file data. 176000 of 3200000 done, 47 hours ETA. A rough calculation of its current rate of 20 entries per second gives a 43 hour ETA.

So much for a best-case scenario.

MJP fucked around with this message at 16:59 on Jun 12, 2016

SyNack Sassimov
May 4, 2006

Let the robot win.
            --Captain James T. Vader


MJP posted:

This was done by me, with no knowledge of the consequences, about a month or two ago.

I now know better. Don't futz with a SAN, call the vendor if you're not sure, even if it looks intuitive and carries no obvious consequences.

I have a tape restore going to another location as a fallback measure. The company culture is tolerant of mistakes, so long as you learn from them, and my boss was OK with me making this test today. Nobody works on the weekends so we at least have 21 hours to try to mitigate damage.

I mean, I wouldn't say don't futz with a SAN as a blanket warning. That kind of thing leads to not learning valuable knowledge about critical bits of IT infrastructure. You do need to do a lot of research though before operating SANs, because when it comes to storage there are a lot of little gotchas that something the SAN technically allows you to do might not be in fact a great idea. It seems fantastic to have a shared volume that multiple machines can access, but you need a filesystem on that volume built to handle multiple initiators hitting it, like VMFS is, whereas NTFS does not generally react well to that.

At the very least it sounds like it's going to be a bad teaching experience rather than a new job experience, so that's good. But yeah, while you should have a healthy respect for what can go wrong in administering a SAN that doesn't mean you should just never touch one unless you want to be one of those guys who has decided he's learned all he needs to know (i.e. a useless lump who will need to find a big company he can hide in without anyone noticing his lack of advancement).

That said, mounting the same NTFS volume to two different machines (both with read/write permissions) is, uh, like #1 on the list of things not to do. Sounds like you're finding out why.

MJP
Jun 17, 2007

Are you looking at me Senpai?

Grimey Drawer

Potato Alley posted:

I mean, I wouldn't say don't futz with a SAN as a blanket warning. That kind of thing leads to not learning valuable knowledge about critical bits of IT infrastructure. You do need to do a lot of research though before operating SANs, because when it comes to storage there are a lot of little gotchas that something the SAN technically allows you to do might not be in fact a great idea. It seems fantastic to have a shared volume that multiple machines can access, but you need a filesystem on that volume built to handle multiple initiators hitting it, like VMFS is, whereas NTFS does not generally react well to that.

At the very least it sounds like it's going to be a bad teaching experience rather than a new job experience, so that's good. But yeah, while you should have a healthy respect for what can go wrong in administering a SAN that doesn't mean you should just never touch one unless you want to be one of those guys who has decided he's learned all he needs to know (i.e. a useless lump who will need to find a big company he can hide in without anyone noticing his lack of advancement).

That said, mounting the same NTFS volume to two different machines (both with read/write permissions) is, uh, like #1 on the list of things not to do. Sounds like you're finding out why.

Yeah, my real concern here is that this is the second major file server issue we've had in six months. In late January, the physical file server died hard, and it was also our backup server. Because Arcserve software is hot garbage at best, any attempt to restore the database from any tape media we had just failed outright. We eventually just hoped for the best and ran restores directly from the media, eventually landing on one that worked. We were basically dead in the water for 2.5 days with my boss and me trading 24-hour days to get things rolling.

Said boss is a totally chill guy and will never come down on someone for making a mistake if they can fix it or put it in the process of fixing, and the company culture is tolerable, but our backups have had one problem after another ever since. I've opened something like 15-20 tickets with Arcserve since early February, once the data was restored, to try to fix individual issues with jobs, media, etc.

Our Friday job ended early due to the backup server crashing due to a known bug, which given the length of backup runtimes and the fact that Dell wanted us to reformat and reinstall the OS, etc. to fix, we simply had no time to pursue in a production environment. I couldn't make a VM for a backup server because until last month we didn't have the appropriate SAS card in the Hyper-V host. Given the Friday crash, I have to drive into NYC after I eat my lunch to load up Wednesday's tapes (Thursday's were pulled for a litigation hold).

It may be a few hours of downtime tomorrow AM at the worst, at least as far as I can tell.

Also I've taken a few notes on how chkdsk proceeds, for the curious:

code:
12:04 PM: 221791 of 3274224 done, 6% complete, 3% total, ETA 46 hrs, appx 30 per sec
12:08 PM, 236286, 7% complete, 3% total, ETA 47 hours, appx 30/sec
12:24 PM, 284200, 8% complete, 4% total, ETA 48 hours, appx 30/50 per sec
Not sure if that 3274224 is megs or what, I think it's roughly the size of the volume (3.2ish TB).

pr0digal
Sep 12, 2008

Alan Rickman Overdrive
I remember when I broke my first SAN (an Apple Xsan). I propagated permissions from the root of the volume and proceeded to lock every system out of the SAN.

I also shut down a secondary controller the other day by accident but since it was the secondary nothing actually broke.

Adbot
ADBOT LOVES YOU

ilkhan
Oct 7, 2004

You'll be sorry you made fun of me when Daddy Donald jails all my posting enemies!

neogeo0823 posted:

The day it happened, I walked into the bank and spoke with 3 different people. Each one telling me there was "nothing they could do" to reverse any of the fees. I politely closed my accounts and went to a credit union. Apparently Citizen's Bank got hit with a class action lawsuit recently for pulling that poo poo for years. About loving time.
Pretty sure that practice was mostly standard across the industry and outlawed in the banking reform that went through a few years ago.

MJP posted:

A clusterfuck came in...
Imagine 2 blind people writing on a chalkboard. At the same time. The problem should be obvious. No bueno.

ilkhan fucked around with this message at 20:30 on Jun 12, 2016

  • Locked thread