Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
Professionally, if it's not backed up or on some form of durable storage, I consider it not essential to the business and will rebuild it completely from scratch instead of trying to recover anything. Case in point - awkward SQL run on a database today outside production means I'm deleting the database completely and restoring from a previous snapshot where I know it was good. Trying to actually recover it all would have taken me way too much time compared to doing the database equivalent of "have you tried turning it off and on again?"

Adbot
ADBOT LOVES YOU

ChiralCondensate
Nov 13, 2007

what is that man doing to his colour palette?
Grimey Drawer

Atomizer posted:

Jesus christ why do you have to make everything so difficult?
He's (successfully) trolling the poo poo out of us.

dexefiend
Apr 25, 2003

THE GOGGLES DO NOTHING!
Have you heard of our Lord and Savior, the ignore feature?

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.

Atomizer posted:

Jesus christ why do you have to make everything so difficult? :psyduck: RAID is not backup, it's for redundancy and uptime, period. You know what's backup? An actual backup solution! You have a separate HDD back up your stuff and/or a remote solution; the number of backups depends on the value and replaceability of the data. Use FreeFileSync, for example, for a quick/free/easy solution to mirror drive A to backup drive B (which you then take offline until the next time you need to update the backup.) Use a service like Backblaze to remotely backup appropriate stuff.

Jesus Christ why do you have to be such an obtuse prick? :psyduck: There's nothing "confusing" about what I am asking. I have 15 hard drives, I'm not buying 15 more for thousands of dollars just to copy and paste everything and store it in a loving bunker. I was just asking if I can get away with only needing to buy 2 or 3 extra empty drives to have some sort of protection if a single drive suddenly dies. I've never had more than 1 drive in a loving year die in my 32 years on this planet, let alone five in a day. I'll just not bother doing RAID or backing anything up, then; thanks for the help as always! :rolleyes:

Devian666 posted:

A number of us have been there when the drive controller has failed and destroyed a raid 5 array. Risk of data loss is not just the probability of individual drive failure. There are far more risks which is why people suggest having a backup.

In the end it's up to you. If you lose all the data and you won't care then just use a raid 5. If there is data you care about keep a copy on a separate device.

Best practice is always 3-2-1 backups, 3 copies, 2 on-site and 1 off-site.

Yeah the data is not vital. It would be incredibly annoying to have to dig and find every episode of The Simpsons again and re-rip my entire DVD collection but it's not stuff I can't get back. Absolutely not worth thousands backing up, but definitely worth buying an extra drive or two if that would keep me safe if one drive died.

I already have important data (family photos and videos, financial stuff) on several hard drives. I just read about RAID and how you only need like 50 TB total for 40 TB of used space and should be safe if a disk ever dies, but if it's possible the RAID5 setup actually increases my chance of data loss (it seems like you are saying if my RAID card dies then I can lose literally everything?) then I obviously would not want that.

Zorak of Michigan posted:

Even then, even if everyone else fails to talk you out of this, at least use RAID 6. 5 just doesn't cut it anymore.

I looked into RAID6 and it seems like you can only use RAID hardware cards with it and it does not support USB drives, or else I definitely would have done that.

frh fucked around with this message at 20:28 on Mar 15, 2019

Droo
Jun 25, 2003

WWWWWWWWWWWWWWWWWW posted:

Jesus Christ why do you have to be such an obtuse prick? :psyduck: There's nothing "confusing" about what I am asking. I have 15 hard drives, I'm not buying 15 more for thousands of dollars just to copy and paste everything and store it in a loving bunker. I was just asking if I can get away with only needing to buy 2 or 3 extra empty drives to have some sort of protection if a single drive suddenly dies. I've never had more than 1 drive in a loving year die in my 32 years on this planet, let alone five in a day. I'll just not bother doing RAID or backing anything up, then; thanks for the help as always! :rolleyes:

You are seriously 32 years old and wrote this?

Anyway, you don't seem to understand the process involved in building a raid array and it seems pretty likely that you would do something stupid to it and destroy all your data at once even if you did manage to successfully create one, so my advice would be to not bother with raid and just keep your giant hoard of porn on separate USB hard drives like it is now.

xzzy
Mar 5, 2009

raid's main function is improving uptime. The ability to lose a disk and not lose data makes it a popular feature but it's not smart to treat it like data protection. You certainly can do it that way (and you'd be far from the first) but someday it will bite back and there will be data loss. Yes, it's very rare. We have thousands of hard drives where I work using a mix of raid 5 or 6 and we've had data loss twice in 15 years.

So if you have priceless data you absolutely must have another copy somewhere.

2 or 3 spares might be enough to get you to the heat death of the universe, but it might also explode into a fireball tomorrow. It's up to you to decide how much risk you want to live with. :shrug:

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.

xzzy posted:

raid's main function is improving uptime. The ability to lose a disk and not lose data makes it a popular feature but it's not smart to treat it like data protection. You certainly can do it that way (and you'd be far from the first) but someday it will bite back and there will be data loss. Yes, it's very rare. We have thousands of hard drives where I work using a mix of raid 5 or 6 and we've had data loss twice in 15 years.

So if you have priceless data you absolutely must have another copy somewhere.

2 or 3 spares might be enough to get you to the heat death of the universe, but it might also explode into a fireball tomorrow. It's up to you to decide how much risk you want to live with. :shrug:

I would like the latter, yes. If my house burns down or my computer explodes I will live with having to find Super Mario Bros The Movie (1993).mkv again as I will have a lot of bigger problems to worry about anyway

I literally know it's not a backup solution. I just would like a thing that helps if one drive dies because of my dumbass cat or because I was stupid and bought a Seagate. I am well aware 2-15 drives can die at one time and I do not care. If I can set something up where 1 drive dies and I pop in another one of identical space and nothing gets lost that would be loving awesome.

xzzy
Mar 5, 2009

Then run raid 5 or software raid or unraid or whatever the heck you want, it's your rig so do whatever you want.

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.

xzzy posted:

Then run raid 5 or software raid or unraid or whatever the heck you want, it's your rig so do whatever you want.

Well, what's what I was asking for advice on lol

Volguus
Mar 3, 2009
What I haven't been able to see here in the last couple of pages is people telling you the main reason to not use raid 5: when a drive dies and you put in a new one the odds are very high for one of the other drives to die while rebuilding the array, therefore causing a total data loss.
However, as you said that the data is not important, so it's up to you how you wanna do it. Personally I think some suggestions here were over the top, but to each his own. What's important back up to the cloud. What's not can stay on premises, but you're better off with plain separate drives than raid5. At least then only the data of one drive dies when it dies.

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.

Volguus posted:

What I haven't been able to see here in the last couple of pages is people telling you the main reason to not use raid 5: when a drive dies and you put in a new one the odds are very high for one of the other drives to die while rebuilding the array, therefore causing a total data loss.
However, as you said that the data is not important, so it's up to you how you wanna do it. Personally I think some suggestions here were over the top, but to each his own. What's important back up to the cloud. What's not can stay on premises, but you're better off with plain separate drives than raid5. At least then only the data of one drive dies when it dies.

OK if raid5 is actually worse than what I'm doing (which is nothing) I will just keep my setup the way it is. Thank you for the info!

Zorak of Michigan
Jun 10, 2006


With all the goodwill in the world, your use case appears to be so unusual that there's not going to be an efficient solution for you that doesn't require you to do a deep dive into the arcana of hard drive storage and giving yourself a second job maintaining your setup.

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.
I'm honestly pretty surprised that worrying about one drive dying at a time is a unique edge case for a home user

astral
Apr 26, 2004

WWWWWWWWWWWWWWWWWW posted:

I'm honestly pretty surprised that worrying about one drive dying at a time is a unique edge case for a home user

That's not the part that makes your case unusual.

Zorak of Michigan
Jun 10, 2006


WWWWWWWWWWWWWWWWWW posted:

I'm honestly pretty surprised that worrying about one drive dying at a time is a unique edge case for a home user

It is when you need to mix USB and internal drives. Most people who need lots of storage and want plug-and-play just buy a Synology or similar device and throw some drives in.

Internet Explorer
Jun 1, 2005





You're asking to swim upstream. This thread is a great resource, but it is a combination of hobbyists and professionals, who while they don't mind rolling their sleeves up, at least take some amount of pride in what they are doing. Most of us are in this thread to help learn and spread best practices. If people are telling you what you are doing is not a good idea, then maybe take the hint? I guess what I'm saying is, none of us would do what you're talking about for ourselves and we certainly aren't going to enjoy walking someone through a booby-trapped obstacle course over text.

It's like a friend that comes and asks you for technical advice and then completely discards everything. It's you. You're being that friend.

phosdex
Dec 16, 2005

If you're already running 15 drives without any redundancy, it would seem rather daunting to make that leap. But you don't have to immediately jump to fully-mirroring those 15.

8-bit Miniboss
May 24, 2005

CORPO COPS CAME FOR MY :filez:
At least with Unraid or MergerFS+Snapraid or Stablebit Drivepool, you'd only lose data on the hard drive that failed rather than the entire array.

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.

8-bit Miniboss posted:

At least with Unraid or MergerFS+Snapraid or Stablebit Drivepool, you'd only lose data on the hard drive that failed rather than the entire array.

Yeah that's what I have right now haha

A coworker of mine was telling me "man if you have that many drives you gotta set up a raid! That drive your stupid rear end dropped would still have its data floating around and it would automatically rebuild when you put the replacement in!" even though I told him a lot of them are USB

It seemed like a cool way to only have to spend another $300 (on two new 8tb drives or something) to make sure I never lost a drive like that again but I guess he and I were mislead.

And I know my setup is weird. Most people do use some sort of server box Linux setup thing but I turned my gaming pc into a server because I'm more familiar with Windows, plus I have a bunch of stuff that took me forever to setup on my server like the ps4 hack host, sonarr, couch potato, etc that I just got deeper and deeper in the hole of never being able to reasonably switch out of a Windows box unless I wanted to take off three months from work to set it all up or something

astral
Apr 26, 2004

WWWWWWWWWWWWWWWWWW posted:

It seemed like a cool way to only have to spend another $300 (on two new 8tb drives or something) to make sure I never lost a drive like that again but I guess he and I were mislead.

Shucked, I hope?

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.

astral posted:

Shucked, I hope?

haha I shucked a few. Once my tower could not hold any more HDs I just left them in the USB enclosure

Raldikuk
Apr 7, 2006

I'm bad with money and I want that meatball!

WWWWWWWWWWWWWWWWWW posted:

haha I shucked a few. Once my tower could not hold any more HDs I just left them in the USB enclosure

Snapraid is basically mare for your situation. It is ideal for archival storage and it keeps all of your data independently intact (no striping) so you can do individual drive recovery while also allowing you to set up parity drives for recovery. It works fundamentally by keeping a list of all your files as well which you store on each drive. So in your prior situation you would have easily been able to retrieve those file names...and probably would have been able to recover the data. The main pitfall of it is that it doesn't calculate the parity realtime and requires you to run that command via a chronjob or manually. It also doesn't do snapshots. That's why it is best suited for archival situations. I use it for my plex server since I only add new content a few times a month.

Corb3t
Jun 7, 2003

Setup an Unraid trial. It's easy and convenient and there's a million great video guides from spaceinvader one on YouTube. I just swapped out a filled 3 TB Seagate (ticking time bomb) with an empty 10 TB and it's rebuilding the data right now without a hitch. It runs Plex, Sonarr, Radarr. I can setup cloud backup with a reverse proxy. I can VPN into it. Once I upgrade, I'm going to setup a Windows vm and game with it.

I really couldn't be happier with my Unraid experience over the past 6 months. The drives can be a bit slow, but it's worth the trade-off.

Corb3t fucked around with this message at 05:15 on Mar 16, 2019

Matt Zerella
Oct 7, 2002

Norris'es are back baby. It's good again. Awoouu (fox Howl)

Gay Retard posted:

Setup an Unraid trial. It's easy and convenient and there's a million great video guides from spaceinvader one on YouTube. I just swapped out a filled 3 TB Seagate (ticking time bomb) with an empty 10 TB and it's rebuilding the data right now without a hitch. It runs Plex, Sonarr, Radarr. I can setup cloud backup with a reverse proxy. I can VPN into it. Once I upgrade, I'm going to setup a Windows vm and game with it.

I really couldn't be happier with my Unraid experience over the past 6 months. The drives can be a bit slow, but it's worth the trade-off.

Pick up 2 ssds and set up a mirrored cache. That'll speed things up. Or use one for cache and pass through the other to the VM as the OS drive.

Atomizer
Jun 24, 2007



WWWWWWWWWWWWWWWWWW posted:

Jesus Christ why do you have to be such an obtuse prick? :psyduck: There's nothing "confusing" about what I am asking. I have 15 hard drives, I'm not buying 15 more for thousands of dollars just to copy and paste everything and store it in a loving bunker. I was just asking if I can get away with only needing to buy 2 or 3 extra empty drives to have some sort of protection if a single drive suddenly dies. I've never had more than 1 drive in a loving year die in my 32 years on this planet, let alone five in a day. I'll just not bother doing RAID or backing anything up, then; thanks for the help as always! :rolleyes:

Yeah the data is not vital. It would be incredibly annoying to have to dig and find every episode of The Simpsons again and re-rip my entire DVD collection but it's not stuff I can't get back. Absolutely not worth thousands backing up, but definitely worth buying an extra drive or two if that would keep me safe if one drive died.

I already have important data (family photos and videos, financial stuff) on several hard drives. I just read about RAID and how you only need like 50 TB total for 40 TB of used space and should be safe if a disk ever dies, but if it's possible the RAID5 setup actually increases my chance of data loss (it seems like you are saying if my RAID card dies then I can lose literally everything?) then I obviously would not want that.

I looked into RAID6 and it seems like you can only use RAID hardware cards with it and it does not support USB drives, or else I definitely would have done that.

I'm totally a prick, but you're being the problem here and I'm not the only one who's noticed. Between the "hey guys I have important data that I didn't bother to backup and I dropped the drive, help me figure out a painful way to recover the contents" and "why isn't RAID backup?!?" there's a pattern of you being in over your head, not knowing what you're doing, and not being willing to accept the advice given. I'm trying to help, but there's a foundation of knowledge missing that would help you to understand all of this.

I don't know the best solution for your situation. Do you have 15 relatively small HDDs? Could you buy a few larger drives to consolidate your data, then use the smaller ones as the backups? (Or vice-versa, I'm actually going to ask a question about this.) Otherwise, you're still responsible for backing up your important data but that doesn't mean you need to back up everything that's on those 15 drives. You might indeed be able to get away with 2-3 more drives for the important backups.

You could totally use RAID5 - which again is for redundancy; you'd still want to have an actual backup of the RAID's contents though. The point of the RAID is to keep the files accessible, not to replace a full backup.

You could try something like Windows Storage Spaces, which is kind of like software RAID but can make sure copies of important files reside on different drives in case one dies. It's still not a real backup solution (where you'd take the backup drive(s) offline and/or have offsite backups) but it's along the lines of what you're trying to do and it works with external drives. Note that the Windows installation could be a point of failure (imagine getting hit with crypto-ransomware and having all your data on the attached drives be encrypted and inaccessible to you.) I think you can actually move the array to a different Windows installation though: I've only tried this once on 3 throwaway drives in my backup Shuttle XPC, but when I took the drives out and moved one to another system, Windows prompted me for the other drives because it identified the storage space to which that one drive belonged. YMMV, and I'm not sure if the drives have to be empty at first when you go to add them to a Storage Space.

WWWWWWWWWWWWWWWWWW posted:

I would like the latter, yes. If my house burns down or my computer explodes I will live with having to find Super Mario Bros The Movie (1993).mkv again as I will have a lot of bigger problems to worry about anyway

I literally know it's not a backup solution. I just would like a thing that helps if one drive dies because of my dumbass cat or because I was stupid and bought a Seagate. I am well aware 2-15 drives can die at one time and I do not care. If I can set something up where 1 drive dies and I pop in another one of identical space and nothing gets lost that would be loving awesome.

This I could definitely help you with: if your copy of "Super Mario Bros The Movie (1993).mkv" is lost, I have a backup! :imunfunny:

WWWWWWWWWWWWWWWWWW posted:

Yeah that's what I have right now haha

A coworker of mine was telling me "man if you have that many drives you gotta set up a raid! That drive your stupid rear end dropped would still have its data floating around and it would automatically rebuild when you put the replacement in!" even though I told him a lot of them are USB

It sounds like your friend has heard of RAID, but doesn't understand it or its limitations fully.

HalloKitty
Sep 30, 2005

Adjust the bass and let the Alpine blast

WWWWWWWWWWWWWWWWWW posted:

Well, what's what I was asking for advice on lol

What you genuinely want is an UnRAID box. It doesn't increase your chances of losing data, unlike normal RAID, because it doesn't affect the data on each drive. It builds parity on a separate drive, and with single parity, you can survive any single drive failure. Double parity.. two. But even if more drives fail, you only lose the data on the failed drives, unlike traditional RAID schemes.

I would suggest it's the absolute best way to protect your data from drive failure without adding risk, and without buying any extra drives. I really don't care if you don't want to back up, that's up to you.

HalloKitty fucked around with this message at 10:10 on Mar 16, 2019

BlankSystemDaemon
Mar 13, 2009



8-bit Miniboss posted:

At least with Unraid or MergerFS+Snapraid or Stablebit Drivepool, you'd only lose data on the hard drive that failed rather than the entire array.
Why are people using it if losing a drive means losing data? As far as I know, the above describes drive concatenation and is not any kind of RAID.

Atomizer posted:

You could totally use RAID5 - which again is for redundancy
You said this before but last time you also said it's for uptime so I didn't crawl down your throat about it.
RAID is only about redundancy when you're using nested levels of mirrors or striping with distributed parity. If there's any device striping without distributed parity, you're doing it for availability and speed.
Put another way: RAID 11, 15, 16, 17, 51, 55, 61, 65, 66, 71, 75, 76, or 77 provides you with redundancy, everything else is just for availability and/or speed.

HalloKitty posted:

What you genuinely want is an UnRAID box. It doesn't increase your chances of losing data, unlike normal RAID, because it doesn't affect the data on each drive. It builds parity on a separate drive, and with single parity, you can survive any single drive failure. Double parity.. two. But even if more drives fail, you only lose the data on the failed drives, unlike traditional RAID schemes.
What you're describing is RAID4, which provides uneven wearing on your parity drive(s) so that drive is more likely to fail.
But nobody except the creator knows how it works, and they aren't saying.

BlankSystemDaemon fucked around with this message at 12:29 on Mar 16, 2019

sharkytm
Oct 9, 2003

Ba

By

Sharkytm doot doo do doot do doo


Fallen Rib

ChiralCondensate posted:

He's (successfully) trolling the poo poo out of us.

I've been saying that silently in my head for weeks. It's a successful troll, considering the number of responses and vitriol they've managed to accumulate.

HalloKitty
Sep 30, 2005

Adjust the bass and let the Alpine blast

D. Ebdrup posted:

What you're describing is RAID4, which provides uneven wearing on your parity drive(s) so that drive is more likely to fail.
But nobody except the creator knows how it works, and they aren't saying.

I know we've had back and forth about this before, and although it resembles RAID4, it isn't 100% like that, in the sense that the data drives don't have to block-level prepared to be a part of the array. Anyway, completely regardless of what your opinions might be, if you don't want to spend money on an extra copy (a backup), then a scheme in which you increase risk of failure (traditional RAID where blocks are striped across the drives) vs a setup where you have parity or double parity data on separate drives, but the individual data drives can be removed at will and used as standard drives, then it would seem to me that the latter is the better solution for someone too cheap to have a backup. I really don't understand how that's a controversial opinion.. traditional RAID setups including for example RAIDZ2 can be great, but with ever-increasing drive sizes, the chances of losing your data increase drastically under a rebuild. If you don't want to have a backup, and that's exactly what the guy did not want to do then a setup in which risk of failure does NOT increase over running individual drives (when drives start to fail under rebuilds) has to be the statistically superior choice

Edit: yes, maybe nobody knows exactly how UnRAID works, but my point is that it doesn't matter, because you can decide to say "gently caress UnRAID" and pull all those drives and use them as standard XFS or BTRFS drives in another linux setup... they're just normal drives without any special structure on them...
I'm simply pointing out an option, that could fit with his need case: to minimise data loss to at worst, only dead drives, to not spend any cash on drives, and to not have a backup solution, but at the same time, provide better protection than no parity at all.

Is it silly that he doesn't want to back anything up? Yeah, but that's up to him.

HalloKitty fucked around with this message at 13:21 on Mar 16, 2019

BlankSystemDaemon
Mar 13, 2009



HalloKitty posted:

I know we've had back and forth about this before, and although it resembles RAID4, it isn't 100% like that, in the sense that the data drives don't have to block-level prepared to be a part of the array. Anyway, completely regardless of what your opinions might be, if you don't want to spend money on an extra copy (a backup), then a scheme in which you increase risk of failure (traditional RAID where blocks are striped across the drives) vs a setup where you have parity or double parity data on separate drives, but the individual data drives can be removed at will and used as standard drives, then it would seem to me that the latter is the better solution for someone too cheap to have a backup. I really don't understand how that's a controversial opinion.. traditional RAID setups including for example RAIDZ2 can be great, but with ever-increasing drive sizes, the chances of losing your data increase drastically under a rebuild. If you don't want to have a backup, and that's exactly what the guy did not want to do then a setup in which risk of failure does NOT increase over running individual drives (when drives start to fail under rebuilds) has to be the statistically superior choice

Edit: yes, maybe nobody knows exactly how UnRAID works, but my point is that it doesn't matter, because you can decide to say "gently caress UnRAID" and pull all those drives and use them as standard XFS or BTRFS drives in another linux setup... they're just normal drives without any special structure on them...
I'm simply pointing out an option, that could fit with his need case: to minimise data loss to at worst, only dead drives, to not spend any cash on drives, and to not have a backup solution, but at the same time, provide better protection than no parity at all.

Is it silly that he doesn't want to back anything up? Yeah, but that's up to him.
The reason why it's bad to use RAID4 even if you're doing it at a file level rather than a block or byte level (the latter being a RAID3) is the same reason that ZFS supports up to three disks worth of failure with RAIDz3: Rebuilds take a long time when you have multi-terabyte disks and the bandwidth has remained substancially unchanged while disk capacity has grown at a rate somewhere between linear and exponential.
So not only are you putting more stress on the parity drive during the normal operation of the array, you're putting more stress on the parity drive when it comes time to rebuild.
Unless I'm much mistaken, UnRAID also doesn't know where on the disk the actual data is, so in addition to increasing the time it takes to rebuild the array, it also makes it unlilely that the data you care about are on the drives that are still working (with ZFS, you can set ditto-blocks to 2 or 3 to increase the number of copies of records, on a per-dataset level).

Nothing is going to save people who don't want to take backups, not even a kind of mainframe/RAS-level RAID with multiple controllers and multipathing.

BlankSystemDaemon fucked around with this message at 13:55 on Mar 16, 2019

eames
May 9, 2009

I think many of these arguments boil down to the problem that the vast majority of home users still think RAID replaces a backup when in reality they'd all be so much better off running JBOD with checksumming/scrubbing and a solid backup strategy. restic, rclone, borgbackup, arq and veeam make this relatively these days.

Unraid's "JBOD with parity" or "RAID4 without striping" scheme is novel but doesn't support checksumming. There are third party programs from community authors that emulate this but... eh, yeah. I can see how it makes sense for some use cases though.

HalloKitty
Sep 30, 2005

Adjust the bass and let the Alpine blast

D. Ebdrup posted:

UnRAID also doesn't know where on the disk the actual data is, so in addition to increasing the time it takes to rebuild the array.

Nothing is going to save people who don't want to take backups, not even a kind of mainframe/RAS-level RAID with multiple controllers and multipathing for an array with RAID666.

I'll address the first point: when you create a share on UnRAID, you can define at which "level" the data is split over the drives.
For example, if I have a share, "Movies", and I decide that it can split one level below that, then all the subfolders and files in my "Movies" share will spread across drives, but NOT anything in those subfolders - they will always remain on the same drive as the parent subfolder. You can also choose to simply allow it to split to any level, in which case UnRAID balances usage between drives automatically (based on an algorithm of your choosing), but the folder structure always remains intact, on each drive.

So you can basically just grab all those drives and merge the folders together, and then you have your original structure.

(For example, if it's just allowed to split one level.. this can be changed per-share)
Movies (root folder, SMB share, this will be created on each drive as necessary)
> Terrible Movie (2001) -- placed on drive 1
>> Extras -- also placed on drive 1, inside the Terrible Movie folder, exactly as you'd expect
> Crap Movie (2006) -- potentially placed on drive 2

To address the second point: I'm not going to disagree, but I'm simply saying that not all solutions are the same, and for someone who doesn't want to take backups or spend too much money, there will be ways to reduce risk. Yeah, if he gets a virus or starts deleting files, something like a snapshotting file system could save his rear end, but he's made it clear he couldn't give a crap about that, so I don't think it's worth spending time on.

If he was keeping backups, we wouldn't be having this conversation in the first place, because almost any RAID6+ like solution would be a great first choice for keeping his data available, and he could revert to his separate backup copy in the case of disaster. Which of course would be the sane thing to do.

HalloKitty fucked around with this message at 14:14 on Mar 16, 2019

BlankSystemDaemon
Mar 13, 2009



eames posted:

Unraid's "JBOD with parity" or "RAID4 without striping" scheme is novel but doesn't support checksumming. There are third party programs from community authors that emulate this but... eh, yeah. I can see how it makes sense for some use cases though.
RAID3 is byte level, RAID4 is block level. Neither involves striping and both have a dedicated parity drive.
Whatever UnRAID is, it matches closest to RAID4 because RAID3 also requires lockstepped rotation (ie. that drives spin at the same rates).
What do you mean by emulating checksumming? Either something is checksummed, or it's not - the only way we can know is to have other people audit the code.

Kirk McKusick is working on adding checksumming to UFS2 in FreeBSD, and it's some of the most complex filesystem code I've seen outside of ZFS.

HalloKitty posted:

I'll address the first point: when you create a share on UnRAID, you can define at which "level" the data is split over the drives.
For example, if I have a share, "Movies", and I decide that it can split one level below that, then all the subfolders and files in my "Movies" share will spread across drives, but NOT anything in those subfolders - they will always remain on the same drive as the parent subfolder. You can also choose to simply allow it to split to any level, in which case UnRAID balances usage between drives automatically (based on an algorithm of your choosing), but the folder structure always remains intact, on each drive.

So you can basically just grab all those drives and merge the folders together, and then you have your original structure.

(For example, if it's just allowed to split one level.. this can be changed per-share)
Movies (root folder, SMB share, this will be created on each drive as necessary)
> Terrible Movie (2001) -- placed on drive 1
>> Extras -- also placed on drive 1, inside the Terrible Movie folder, exactly as you'd expect
> Crap Movie (2006) -- potentially placed on drive 2

To address the second point: I'm not going to disagree, but I'm simply saying that not all solutions are the same, and for someone who doesn't want to take backups or spend too much money, there will be ways to reduce risk. Yeah, if he gets a virus or starts deleting files, something like a snapshotting file system could save his rear end, but he's made it clear he couldn't give a crap about that, so I don't think it's worth spending time on.

If he was keeping backups, we wouldn't be having this conversation in the first place, because almost any RAID6+ like solution would be a great first choice for keeping his data available, and he could revert to his separate backup copy in the case of disaster. Which of course would be the sane thing to do.
The use of "potentially" in your example makes me think this is more theory than practice. Have you looked at the data structure on-disk to verify that this is how it works?
Is there an algorithm for, for examples, your documents folder that spreads that information across every single disk?

Traditional RAID implementations also suffer from write-holes unless you take a leaf out of databases and use a write-ahead logs - does UnRAID do anything to solve this problem?
Given what eames said about there only being community authored versions trying to implement checksumming, that means that there's also no way to know the state of the data on-disk, is there?
FreeBSDs GEOM RAID3 implementation provides XOR verification, and GEOM journaling can go on top of that (and checksumming in UFS2 can be on top of that, once it's ready).

If someone doesn't want to spend much money on backup or doing janitorial computer touching, there's really only one option: Upload their many terabytes to a backup provider (at a few hundred kilobytes per second because SLAs never guarentee bandwidth).

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
Services like Backblaze B2 or perhaps Tarsnap are much cheaper than the usual cloud service providers like Google, Amazon, Microsoft, or even Apple because when you retrieve data back outside their cloud you get charged a metric ton. But really, if there’s something that could be replaced fairly easily don’t pay to back it up. If something’s easy to get now and can change in the future, that’s another matter. The other icky part of backups is that historically backups could also rot and fail so you’d want to test backups frequently too, but who the hell does that at home? Hence, super solid offsite storage with really low byte level failure rates help guard against risks there.

Me, I generate a lot of derivative data that takes days and weeks of GPU crunch time to compute so while it could be reconstructed I’d just plain rather not waste the time and money to do it again and that becomes data I should be backing up or I accept the risks of data loss and don’t get emotional if I lose it. But at a certain point I will just do all my compute in GCP and archive it at a fairly cheap rate in the cloud garden and expect to never take it out.

eames
May 9, 2009

D. Ebdrup posted:

RAID3 is byte level, RAID4 is block level. Neither involves striping and both have a dedicated parity drive.

then please be so kind and fix the wikipedia article :)

necrobobsledder posted:

The other icky part of backups is that historically backups could also rot and fail so you’d want to test backups frequently too, but who the hell does that at home? Hence, super solid offsite storage with really low byte level failure rates help guard against risks there.

most modern backup tools do this automatically for you

IOwnCalculus
Apr 2, 2003





It sounds like the main benefit of Unraid is similar to that of Drivepool - in that a URE during a rebuild won't poo poo your entire array in the process, like it would with hardware RAID or mdraid or anything at that level. But you will lose at least some data, and the question then is that loss / corruption detectable and flagged so you know to restore from backup?

I still have everything backed up via Crashplan, but that's mostly because I'm still grandfathered into my old unlimited account that hasn't run out yet. I think I've got a year of $2.50/mo on top of that and that's still way cheaper than any other solution. I'll probably switch to a G Suite account when that runs out.

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.

sharkytm posted:

I've been saying that silently in my head for weeks. It's a successful troll, considering the number of responses and vitriol they've managed to accumulate.

I am not sure why I am coming off as a troll but I apologize if I am. Basically my computer usage has been this since 1994 when I started with Windows 3.1:

1) upgrade my computer/get a new computer and transfer over all of my old data with floppy discs or CD-Rs.
2) when USB hard drives became a thing, I'd dump all my files onto that when upgrading so it did not clutter my new OS/computer
3) In the early 2000s I needed an MP3 and old book report from 1996 and I still had it which began my life of data hoarding
4) as hard drives became cheap I began getting bigger ones and would send them to my cousin in the US who would load them up with movies and games and TV shows that were basically impossible to get
5) I began ripping my own DVD and game collection so I would have instant access to them
6) I found out about MILF porn
7) I gave up on having a gaming PC and just turned my gaming PC into a server (basically a gaming PC with a ridiculous number of hard drives) because of how much more important Plex was to me than sitting in front of gaming PC. However, I still use it as a gaming PC with NVidia Gamestream, so I can leave it tucked away in a basement and never even look at it but still play games on it

I know it's ridiculous to not delete anything but storage is cheap enough now that whenever I fill an old drive I just buy a new one. I buy a new drive maybe once every year, which slowed down a lot now though since 8tb are affordable and it takes forever to fill those.

I'm sorry again if my posts came off as so dumb that people thought it was a troll but I'm just not that bright. I know my setup is probably moronic but I am not familiar with Linux and it's hard for me to change my entire setup the way it is already. Trust me, I'd love some sort of neat and tidy box that takes up minimal electricity instead.

Raldikuk posted:

Snapraid is basically mare for your situation. It is ideal for archival storage and it keeps all of your data independently intact (no striping) so you can do individual drive recovery while also allowing you to set up parity drives for recovery. It works fundamentally by keeping a list of all your files as well which you store on each drive. So in your prior situation you would have easily been able to retrieve those file names...and probably would have been able to recover the data. The main pitfall of it is that it doesn't calculate the parity realtime and requires you to run that command via a chronjob or manually. It also doesn't do snapshots. That's why it is best suited for archival situations. I use it for my plex server since I only add new content a few times a month.

Gay Retard posted:

Setup an Unraid trial. It's easy and convenient and there's a million great video guides from spaceinvader one on YouTube. I just swapped out a filled 3 TB Seagate (ticking time bomb) with an empty 10 TB and it's rebuilding the data right now without a hitch. It runs Plex, Sonarr, Radarr. I can setup cloud backup with a reverse proxy. I can VPN into it. Once I upgrade, I'm going to setup a Windows vm and game with it.

I really couldn't be happier with my Unraid experience over the past 6 months. The drives can be a bit slow, but it's worth the trade-off.

Thank you. I figured there might be some sort of software solution. I have no problem with buying a new 10TB external usb 3.0 HD to use for this purpose and nothing else. I just didn't want to buy a new drive for every drive I already had is all. I am going to look into the options you posted over the next two weeks and hopefully figure it all out.

HalloKitty posted:

What you genuinely want is an UnRAID box. It doesn't increase your chances of losing data, unlike normal RAID, because it doesn't affect the data on each drive. It builds parity on a separate drive, and with single parity, you can survive any single drive failure. Double parity.. two. But even if more drives fail, you only lose the data on the failed drives, unlike traditional RAID schemes.

I would suggest it's the absolute best way to protect your data from drive failure without adding risk, and without buying any extra drives. I really don't care if you don't want to back up, that's up to you.

Yeah I would go with the unRAID box but my experience with Linux pretty much begins and ends with having an Android phone. I am way, way behind the posters here when it comes to computer knowledge. Not to mention I kind of wanted to stick with Windows because I can use it as a gaming PC to Gamestream every room in my house. Though I am willing to give that up if I have to.

Atomizer posted:

I'm totally a prick, but you're being the problem here and I'm not the only one who's noticed. Between the "hey guys I have important data that I didn't bother to backup and I dropped the drive, help me figure out a painful way to recover the contents" and "why isn't RAID backup?!?" there's a pattern of you being in over your head, not knowing what you're doing, and not being willing to accept the advice given. I'm trying to help, but there's a foundation of knowledge missing that would help you to understand all of this.

I don't know the best solution for your situation. Do you have 15 relatively small HDDs? Could you buy a few larger drives to consolidate your data, then use the smaller ones as the backups? (Or vice-versa, I'm actually going to ask a question about this.) Otherwise, you're still responsible for backing up your important data but that doesn't mean you need to back up everything that's on those 15 drives. You might indeed be able to get away with 2-3 more drives for the important backups.

You could totally use RAID5 - which again is for redundancy; you'd still want to have an actual backup of the RAID's contents though. The point of the RAID is to keep the files accessible, not to replace a full backup.

You could try something like Windows Storage Spaces, which is kind of like software RAID but can make sure copies of important files reside on different drives in case one dies. It's still not a real backup solution (where you'd take the backup drive(s) offline and/or have offsite backups) but it's along the lines of what you're trying to do and it works with external drives. Note that the Windows installation could be a point of failure (imagine getting hit with crypto-ransomware and having all your data on the attached drives be encrypted and inaccessible to you.) I think you can actually move the array to a different Windows installation though: I've only tried this once on 3 throwaway drives in my backup Shuttle XPC, but when I took the drives out and moved one to another system, Windows prompted me for the other drives because it identified the storage space to which that one drive belonged. YMMV, and I'm not sure if the drives have to be empty at first when you go to add them to a Storage Space.


This I could definitely help you with: if your copy of "Super Mario Bros The Movie (1993).mkv" is lost, I have a backup! :imunfunny:


It sounds like your friend has heard of RAID, but doesn't understand it or its limitations fully.

Yeah sorry if I came off like a dick, I just get rubbed the wrong way when I feel like I am being talked down to. Sorry. It's funny because everyone I know in real life thinks I am some computer genius when in actuality I don't know 1/20th of what you all know.

I've luckily never been hit with ransomware, and I've never had more than 1 drive die at once, so I will probably just roll the dice and use Windows Storage Spaces or something. Thanks again for the advice. Sorry again if I came off like a moron; if I did it's because I am one. And if anyone thinks I'm a troll because of this avatar, some goober in GBS gave it to me because I said women should be treated equally in a pro-Trump thread :lol:

frh fucked around with this message at 17:38 on Mar 16, 2019

Matt Zerella
Oct 7, 2002

Norris'es are back baby. It's good again. Awoouu (fox Howl)
Use what you want but not knowing Linux really doesn't mean anything when it comes to UnRAID. You can comfortably live in the WebUI and almost never have to touch a command prompt.

Hell the community apps plugin lets you use docker in the same exact way once you wrap your head around directory maps.

Just whatever you do, please for the love of god don't mix internal and USB drives.

frh
Dec 6, 2014

Hire Kenny G to play for me in the elevator.

Matt Zerella posted:


Just whatever you do, please for the love of god don't mix internal and USB drives.

This is an issue even with a software solution like Windows Spaces?

Adbot
ADBOT LOVES YOU

Matt Zerella
Oct 7, 2002

Norris'es are back baby. It's good again. Awoouu (fox Howl)

WWWWWWWWWWWWWWWWWW posted:

This is an issue even with a software solution like Windows Spaces?

Friend. Don't do it. If you're using external drives just use them as JBOD.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply