NAS/Storage Megathread: What is this "File Deletion" You Speak of

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›846 »

AlternateAccount: Apr 25, 2005; FYGM

Avian Pneumonia posted:

Then, I'd like to combine the two hard drives on to only one while again avoiding any duplicates between the two. How would I do this?

Are the paths the same to the duplicate files or does the arrangement differ?

# ? May 23, 2014 15:29

Adbot: ADBOT LOVES YOU

# ? May 16, 2024 10:39

Avian Pneumonia: May 24, 2006; ASK ME ABOUT MY OPINIONS ON CANCEL CULTURE

AlternateAccount posted:

Are the paths the same to the duplicate files or does the arrangement differ?

In many cases, yes. But some things are all over the place.

# ? May 23, 2014 17:28

MMD3: May 16, 2006; Montmartre -> Portland

MMD3 posted:

A few questions from a Synology noob.

1) I tried to move my Google Drive folder over to a new share I created on my Synology. I moved the folder over but for some reason Google Drive won't let me throwing "The location selected for your Google Drive folder is not writable. Please fix this or choose another location for your Google Drive folder." I've ensured that Read/Write access is set up for Guests, is there something I'm missing or is this not possible?

2) For some reason occasionally the shares I've set up on my Synology will show up as offline. Then if I open them via explorer they'll come back online and everything will be fine again. It doesn't happen all that often but it's pretty annoying when I go to stream something via Plex or what not and it tells me it can't find my files.

Any synology owners out there with an answer for either of these questions?

# ? May 23, 2014 17:34

bizwank: Oct 4, 2002

MMD3 posted:

Any synology owners out there with an answer for either of these questions?

2) Do you have HDD hibernation turned on? (Control Panel -> Hardware). I do, and when in hibernation my shares look offline until I open an explorer window, then it takes a couple seconds for them all to reappear. This doesn't cause me any problems though, ie. any device I have accessing the shares seems to be able to wake it up, I never get file not found errors. Are you running Plex on the NAS or on a separate device?

# ? May 23, 2014 20:03

suddenlyissoon: Feb 17, 2002; Don't be sad that I am gone.

MMD3 posted:

Any synology owners out there with an answer for either of these questions?

Part 1 has to do with the way the DSM locks files I'd assume. I run in to that sometimes when my Mac is using a specific folder for a time machine backup.

To add to what Bizwank said to part 2, are you mapping the drives in Windows or are you just using them through network share?

# ? May 23, 2014 21:01

Thanks Ants: May 21, 2004; #essereFerrari

If it's just you using Google Drive then use the Google Drive app package thing, point it at a share and then use that to store your stuff in, then remove the Google Drive application from your PC.

# ? May 23, 2014 21:28

carb0nex: Nov 5, 2012

I am using a Corsair 400r case and Corsair 600w PSU with a pair of moelx->sata power connectors. I am looking to add more drives to my system, but I am running out of power connectors. Is there any sort of chassis that will hold an ATX board (with the stock CPU cooler) and at least ten 3.5'' drives? The only way to get more sata connectors is to buy a biggest PSU, which this system certainly doesn't need.

Perhaps a server chassis of some kind?

# ? May 26, 2014 15:33

Don Lapre: Mar 28, 2001; If you're having problems you're either holding the phone wrong or you have tiny girl hands.

Your case has nothing to do with power connectors on your motherboard. You should just get some molex to sata connectors. You can order custom cables with any number of connectors and any spacing you want.

As far as cases for 10 drives. There are plenty. The fractal Define XL holds 10 drives with a second hard drive holder i think.
Also

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=100007583%20600030566%20600030567%20600285476&IsNodeId=1&name=13

# ? May 26, 2014 16:33

hifi: Jul 25, 2012

carb0nex posted:

I am using a Corsair 400r case and Corsair 600w PSU with a pair of moelx->sata power connectors. I am looking to add more drives to my system, but I am running out of power connectors. Is there any sort of chassis that will hold an ATX board (with the stock CPU cooler) and at least ten 3.5'' drives? The only way to get more sata connectors is to buy a biggest PSU, which this system certainly doesn't need.

Perhaps a server chassis of some kind?

There's also something like this

# ? May 26, 2014 16:59

Ika: Dec 30, 2004; Pure insanity

SamDabbers posted:

Maybe a Powershell solution like one of these?

That looked interesting and I just tried the first version, and modified it to add a filter to ignore certain filetypes to improve hashing performance.

code:

            ForEach-Object {
                if ($_.FullName.ToLower().EndsWith(".mpq") -or
                    $_.FullName.ToLower().EndsWith(".ignore"))
                {
                    continue;
                }

However unleashing it on a folder with just 60k files uses half a gig of memory, so I'm not sure how badly it will blow up if its used on really messy drives. I guess at some point you would want to deduplicate folder names etc which can't be done in a script.

E: inverted logic to simplify modding.

Ika fucked around with this message at 20:24 on May 26, 2014

# ? May 26, 2014 20:21

dorkanoid: Dec 21, 2004

I've mentioned before the success of running Xpenology on the HP N54L; however, I've now upgraded mine to have 8GB RAM and installed ESXi on it. I was wondering if there are any good alternatives for NAS OS that are less "hacky" in the "unsupported" sense and more "hacky" in the "geek out with config files" sense?

As I understand it the "Synology Hybrid Raid" uses regular Linux RAID software; I like the general setup where I can just add disks freely, and would like something similar. Is ZFS the way to go? And how is iSCSI to set up outside the nice bubble of Synology?

# ? May 26, 2014 20:29

Moey: Oct 22, 2010; I LIKE TO MOVE IT

Zfs on linux?

# ? May 26, 2014 20:48

SamDabbers: May 26, 2003

Moey posted:

Zfs on linux?

ZFS on FreeBSD :colbert:

# ? May 26, 2014 22:20

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Except for the part where he said he likes to be able to just add disks freely.

# ? May 26, 2014 22:20

dorkanoid: Dec 21, 2004

Thermopyle posted:

Except for the part where he said he likes to be able to just add disks freely.

That's the important part right now; the current state of the NAS is an assortment of disk sizes - I want to replace them one by one over the coming months and have a somewhat decent NAS while doing so.

Right now I'm using the Xpenology version of DSM 5.0 where iSCSI doesn't work

As far as I can tell, Synology Hybrid Raid works by slicing each disk into lowest common denominator-sized pieces, then setting up some kind of RAID across that - in such a way that any one disk can fail without data loss.

# ? May 26, 2014 22:41

Moey: Oct 22, 2010; I LIKE TO MOVE IT

Thermopyle posted:

Except for the part where he said he likes to be able to just add disks freely.

Oops. Must have skimmed over this earlier.

# ? May 27, 2014 00:34

Cockmaster: Feb 24, 2002

Moey posted:

Avoid Drobo and go with a Synology or roll your own.

What exactly is wrong with the Drobo (for basic consumer applications, that is)? I'm just looking for something to store stuff on and play videos off of (and ideally run a BitTorrent client, for which there are apps on both Synology and Drobo).

# ? May 27, 2014 00:49

berzerkmonkey: Jul 23, 2003

After a power outage the other night that killed my OS HDD, I'm doing some updating and am thinking about a NAS. I've currently got an external HDD for backup, but I'd like something a bit more robust, and not just another single point of failure. I don't have a lot to spend, but I want to find something that is reliable and doesn't consume a lot of power.

I came across the Buffalo Technology BUFFALO LinkStation Pro Duo 2-Bay Diskless Enclosure Network Attached Storage (NAS) - LS-WVL/E at Dell for $90 shipped (Amazon has it for $96 shipped) and was wondering if it is a decent deal. Can anyone comment on the Buffalo software?

I know Synology is the go-to, but it's almost $200 for the DS213j and I can use the difference toward a WD Red drive. Is Synology really that much better, and, if so, why?

berzerkmonkey fucked around with this message at 16:44 on May 28, 2014

# ? May 28, 2014 16:16

Don Lapre: Mar 28, 2001; If you're having problems you're either holding the phone wrong or you have tiny girl hands.

berzerkmonkey posted:

After a power outage the other night that killed my OS HDD, I'm doing some updating and am thinking about a NAS. I've currently got an external HDD for backup, but I'd like something a bit more robust, and not just another single point of failure. I don't have a lot to spend, but I want to find something that is reliable and doesn't consume a lot of power.

I came across the Buffalo Technology BUFFALO LinkStation Pro Duo 2-Bay Diskless Enclosure Network Attached Storage (NAS) - LS-WVL/E at Dell for $90 shipped (Amazon has it for $96 shipped) and was wondering if it is a decent deal. Can anyone comment on the Buffalo software?

I know Synology is the go-to, but it's almost $200 for the DS213j and I can use the difference toward a WD Red drive. Is Synology really that much better, and, if so, why?

I dont know one way or another about Buffalo support. But Synology supports units long after they have left sales channels.

# ? May 28, 2014 18:08

Civil: Apr 21, 2003; Do you see this? This means "Have a nice day".

berzerkmonkey posted:

After a power outage the other night that killed my OS HDD, I'm doing some updating and am thinking about a NAS. I've currently got an external HDD for backup, but I'd like something a bit more robust, and not just another single point of failure. I don't have a lot to spend, but I want to find something that is reliable and doesn't consume a lot of power.

I came across the Buffalo Technology BUFFALO LinkStation Pro Duo 2-Bay Diskless Enclosure Network Attached Storage (NAS) - LS-WVL/E at Dell for $90 shipped (Amazon has it for $96 shipped) and was wondering if it is a decent deal. Can anyone comment on the Buffalo software?

I know Synology is the go-to, but it's almost $200 for the DS213j and I can use the difference toward a WD Red drive. Is Synology really that much better, and, if so, why?

Synology's advantage is software that does a whole lot more than most NAS appliances. Get one if you want to be able to run a webserver, database server, webcam surveillance station, itunes/dlna/plex media server, and a handful of other applications. If you just want to store files, save your money.

# ? May 28, 2014 19:39

berzerkmonkey: Jul 23, 2003

Civil posted:

Synology's advantage is software that does a whole lot more than most NAS appliances. Get one if you want to be able to run a webserver, database server, webcam surveillance station, itunes/dlna/plex media server, and a handful of other applications. If you just want to store files, save your money.

Yeah, I'm pretty much looking to back up files, photos, music, etc. That being said, could you use it for storage AND as a media server, or that kind of thing frowned upon? I assume that you're not going to want to use something as a backup system, then constantly write and delete media files to it. Or is something like that not really an issue as long as you have a RAID setup?

Personally, I'd prefer fewer devices running all the time - the PC I'm using as my media storage isn't really a hog, but I'm sure a NAS is going to use much less power, especially when I'm not streaming from it.

Or, poo poo - should I just take my existing media storage PC, drop in some Red drives, and use FreeNAS?

berzerkmonkey fucked around with this message at 21:42 on May 28, 2014

# ? May 28, 2014 21:25

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Not a NAS question, but a storage one.

I use truecrypt to encrypt my hard drives right now, but am planning to upgrade to a new SSD + traditional hard drive soon. Rather than using truecrypt, I just want to use the self-encrypting drive feature on both. For the SSD I am planning to get a Samsung Evo SSD, but I'm not sure what the best option is for the storage drive. The WD Reds can enable SED, but I can't tell if this is on every WD Red or only for specific models.

# ? May 28, 2014 21:42

BlankSystemDaemon: Mar 13, 2009

It's probably a good thing that you're dropping truecrypt, but it's WD Se (and Re) that supports self-encryptiion if I recall correctly.

# ? May 29, 2014 08:07

Un-l337-Pork: Sep 9, 2001; Oooh yeah...

I have an older machine (E8400, 4GB RAM, nice video card, single Intel NIC onboard) that I want to repurpose as a storage device. The primary use of this will be to backup media (tv/movie/music/photo) and to serve that media, primarily to a large television (DLNA -- seems to play most MP4/AVI that I throw at it. I'm not worried about handling MKV or transcoding at this point).

Technically, I am fairly capable, but my knowledge of storage is skin deep. I understand the basics of RAID/ZFS and can do very simple things like use LVM to resize a volume, but I am not really comfortable with getting my hands dirty in the realm of storage (not that I am opposed to doing so, but it seems complicated and I don't have the time right now to learn all of the ins and outs).

Unless I am strongly advised to do otherwise, I would rather do this project myself and re-purpose my existing device than buy a Synology appliance, as I generally enjoy doing this sort of thing myself.

I'm unsure of xpenology and the various NAS solutions (NAS4FREE etc) -- the documentation seems a bit dodgy and, sadly, I have far more experience with Linux than FreeBSD. If they are the better solutions, I am not opposed to using them.

Aside from running miniDLNA (or whatever) and maybe a torrent client, this box will just be for storage.

My primary concern is stability/recoverability. If a drive fails, I want to be able to take it out, pop another one in, and have the RAID / Filesystem fix itself automatically with a minimum of fuss. I will probably be buying the Seagate NAS drives (or WD reds maybe).

(1) After reading, it seems like I should either be looking at ZFS or RAID with a controller card. Are both of these equally valid for this application? Would ZFS alert me when something is wrong and allow me to replace a drive easily?

(1a) If RAID+Controller card, what off-the-shelf card would you recommend? (I don't want to eBay some older card that can be flashed to be awesome or whatever -- just give me something that I can drop in and have it work).

(2) Would some flavor of Linux be OK for this? Any particular distribution that I should consider or avoid (I actually like ubuntu, but recognize that it is not great for all things).

(2a) If Linux is not good for this, what would you recommend?

Basically, I don't mind putting the work in to get things set up and running -- I just don't want to have to micromanage this thing.

=== UPDATE ===

I'm reading this presentation from the FreeNAS forums (https://drive.google.com/file/d/0BzHapVfrocfwc3VsQ3Y0eldFTHM/edit?usp=sharing) and I'm becoming increasingly convinced that this might be the way to go, at least for storage. Now I'm thinking of putting together a new box for the storage hardware and turning the old one into a front-end media server.

Un-l337-Pork fucked around with this message at 16:47 on May 29, 2014

# ? May 29, 2014 14:34

dorkanoid: Dec 21, 2004

dorkanoid posted:

That's the important part right now; the current state of the NAS is an assortment of disk sizes - I want to replace them one by one over the coming months and have a somewhat decent NAS while doing so.

Right now I'm using the Xpenology version of DSM 5.0 where iSCSI doesn't work

As far as I can tell, Synology Hybrid Raid works by slicing each disk into lowest common denominator-sized pieces, then setting up some kind of RAID across that - in such a way that any one disk can fail without data loss.

I ended up getting Xpenology DSM 5.0 installed on ESXi (after upgrading it to 8GB RAM), which is awesome. Speeds are still basically capping out my Gbit LAN, iSCSI works if you update to "DSM 5.0-4458 update 2" and I have enough resources to run more VMs on the N54L.

I followed this guide, which also explains how to set the disks up in Raw Device Mapping (RDM), so there's no virtual layer down to the disks to steal performance.

The only "problem" is that the disk you set as the ESXi data store can't be RDMed, so effectively you lose that one disk of space - I'm considering moving the data store to an old laptop drive an put it in the CDROM slot.

# ? May 29, 2014 16:57

BlankSystemDaemon: Mar 13, 2009

If you have a small spare SSD lying around, putting your data store on that isn't a bad idea when you're doing RDM in ESXi.

# ? May 29, 2014 18:33

infinite99: Aug 9, 2006; ANY OF YALLS DICKS HARD??

I currently have a file server set up using an old version of Amahi which uses a greyhole set up for storage pooling. It's actually been rock solid and I've tacked on a bunch of other things to it such as Plex, SABNZBd, and Transmission. The only thing I don't like is how transferring files between shares in the storage pool is unnecessarily complicated or I'm just doing something completely wrong.

I have my shares set up on my Windows machines and things get downloaded to one folder and then I'd have to transfer them to another shared folder in the storage pool which seemed like it was probably copying the files to the Windows machine and then transferring it back to the server even though everything is stored on the one server. I think my set up is just completely wrong and I want to scrap the whole thing and redo everything.

The reason I used Amahi was because I had a bunch of differently sized hard drives that I wanted to use all together as a NAS and it's a good solution but using other services on the same machine probably wasn't a great idea. Now I want to buy a bunch of Red Drives and maybe play around with ESXi so I'm going to replace all my current drives with new ones, but what's the best way to back up like 5TB worth of stuff and then replacing the old drives? Would I have to just move everything over to external drives before setting up ESXi and something like XPEnology or FreeNAS and then putting back the data?

My current thinking is having one box with a bunch of virtual machines that will handle one service but they'll all go to the virtual machine that's acting as the NAS. My only concern is with Plex transcoding since i'm currently running a Core 2 Duo in this machine and I'd be serving like 2 streams at once possibly. It might not even need to transcode at that point either so maybe it's alright.

# ? May 29, 2014 22:13

swampface: Apr 30, 2005; Soiled Meat

Does this mb/processor seem like a good combo for a low-cost, multi-purpose NAS/server with ECC/ZFS:

ASUS M5A78LM @ $50.99
AMD FX-4300 @ 109.99

I'm planning it as a roll-my-own ubuntu machine for file/media storage, crashplan, plex(w/transcoding), sickbeard, sabnzbd, zoneminder, and possibly future home-automation stuff.

As far as I can tell that's the cheapest way to get in to the world of ECC memory and ZFS. Does anyone know if there's anything lower cost than that?

# ? May 30, 2014 21:07

BobHoward: Feb 13, 2012; The only thing white people deserve is a bullet to their empty skull

If you're looking to build an ECC server (meaning that you are deciding that data integrity matters to you, and are willing to spend a little extra on it), trying to go super cheap on the motherboard doesn't make a lot of sense to me.

# ? May 30, 2014 21:22

SamDabbers: May 26, 2003

Desktop ASUS AMD boards have excellent support for ECC, even on the low-end. My ZFS setup is a generation or two older (ASUS M4A88TD-M) but it detects the ECC RAM just fine (as being ECC) and there are BIOS knobs for ECC scrub frequency.

# ? May 30, 2014 21:35

swampface: Apr 30, 2005; Soiled Meat

Good point. I just wish there was some space between $50 and $200 motherboards. Though even with a $50 motherboard I figure I'm better off than non-ECC ram and an ext4 software raid, even if I am backing everything up to crashplan.

# ? May 30, 2014 21:35

Comatoast: Aug 1, 2003; by Fluffdaddy

This Lenovo Thinkserver gets down to $300 once a month, occasionally lower. The case is awful but it does definitely exist and will hold five drives (3 3.5 slots, 2 5.25 slots).

The quad core E3 processor is a reasonable upgrade also.

Comatoast fucked around with this message at 01:29 on May 31, 2014

# ? May 31, 2014 00:33

swampface: Apr 30, 2005; Soiled Meat

Wow, that's a pretty good deal. Save me a lot of work too. Thanks!

# ? May 31, 2014 00:43

eddiewalker: Apr 28, 2004; Arrrr ye landlubber

Newegg as a $50 rebate running on the N54L again. $279, so it's been a little cheaper, but this time its for the version with 4gb of ECC rather than 2gb.

Expires tomorrow. http://images10.newegg.com/uploadfilesfornewegg/rebate/SH/Newegg/HP59-108-164_56663May30Jun0114yh62.pdf

# ? Jun 1, 2014 01:25

fookolt: Mar 13, 2012; Where there is power
There is resistance

What's a dead simple solution for RAID0 or RAID5 backups for 2 Windows computers and 2 Mac computers? Synology or something similar? I'd like to pay less than $500 if possible.

# ? Jun 1, 2014 03:06

Megaman: May 8, 2004; I didn't read the thread BUT...

fookolt posted:

What's a dead simple solution for RAID0 or RAID5 backups for 2 Windows computers and 2 Mac computers? Synology or something similar? I'd like to pay less than $500 if possible.

If you want to stay within budget I personally would do N54L with 2 1TB drives in RAID1 with FreeNAS to start, that's literally going to be about $500. RAID5 will cost about 600-700$ because you need a minimum of 3 disks for RAID5, which isn't recommended.

If you care about the data and don't mind going over budget I would do N54L with 4 1tb drives minimum in RAIDZ1/2/3 with FreeNAS depending on how much you care about the data. That will be around $700. That will give you the most flexibility of choice between parity vs usable space.

# ? Jun 1, 2014 03:23

Don Lapre: Mar 28, 2001; If you're having problems you're either holding the phone wrong or you have tiny girl hands.

Amazon has the ds214+ for $299 if you just need 2 drives.

# ? Jun 1, 2014 03:58

modeski: Apr 21, 2005; Deceive, inveigle, obfuscate.

I'm currently speccing out a new NAS and am tossing up between an SSD and SATA drive for the system disk. I like the speed of SSDs, but that'll only be noticeable during installs/updates/reboots etc. My current NAS has lasted about four years and I'd expect this new one to last longer as it's more expandable, so a SATA drive might be more reliable, especially because of the amounts of writing I'll be doing.

Would love to hear people's opinions on this, and/or recommendations for particular drives. I'll be running WHS2011+Drivepool and starting off with 4x4Tb WD RED drives for the storage pool. 128Gb would be fine for SSD, or 500Gb for SATA.

VVV Thanks, I'll bear that all that in mind.

modeski fucked around with this message at 12:06 on Jun 3, 2014

# ? Jun 3, 2014 08:05

Comatoast: Aug 1, 2003; by Fluffdaddy

Just make sure to source the drives one at a time from different places. Otherwise you'll end up with drives from the same batch and it becomes more likely that more than one will fail at a time. I go so far as to source different types of drives for a single server. My server has a seagate nas drive, a WD red and a regular old 7200 rpm Toshiba. Each of them was shipped to me separately. Also, disregard newegg as a potential vendor unless you like your drives being smashed around during shipping.

Based purely on my personal experience, I'd say a spinning disk is more reliable than a ssd. Not only is the mean time before failure longer in a spinning disk but you'll have more warning that its about to poo poo the bed. SSD tend to die with no warning. Two spinning disks in a raid1 is the way to go for the OS disks.

Comatoast fucked around with this message at 12:09 on Jun 3, 2014

# ? Jun 3, 2014 12:02

Adbot: ADBOT LOVES YOU

# ? May 16, 2024 10:39

Varashi: Sep 1, 2006; THE MAN is limiting my BANDWIDTH :argh: [belgian goons]

dorkanoid posted:

I ended up getting Xpenology DSM 5.0 installed on ESXi (after upgrading it to 8GB RAM), which is awesome. Speeds are still basically capping out my Gbit LAN, iSCSI works if you update to "DSM 5.0-4458 update 2" and I have enough resources to run more VMs on the N54L.

I followed this guide, which also explains how to set the disks up in Raw Device Mapping (RDM), so there's no virtual layer down to the disks to steal performance.

The only "problem" is that the disk you set as the ESXi data store can't be RDMed, so effectively you lose that one disk of space - I'm considering moving the data store to an old laptop drive an put it in the CDROM slot.

Using RDMS works, however you're probably better off with passing through a SATA controller to your VM.

That said though, is SSD caching finally working with xpenology?
Last time I tried creating an SSD read/write cache it failed to mount to cache volume.

# ? Jun 3, 2014 12:17

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›846 »