NAS/Storage Megathread: What is this "File Deletion" You Speak of

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›2 »

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Not a NAS question, but a storage one.

I use truecrypt to encrypt my hard drives right now, but am planning to upgrade to a new SSD + traditional hard drive soon. Rather than using truecrypt, I just want to use the self-encrypting drive feature on both. For the SSD I am planning to get a Samsung Evo SSD, but I'm not sure what the best option is for the storage drive. The WD Reds can enable SED, but I can't tell if this is on every WD Red or only for specific models.

# ¿ May 28, 2014 21:42

Adbot: ADBOT LOVES YOU

# ¿ Apr 29, 2024 16:45

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

I want to buy a 4-5 bay NAS. I don't really care if it's a synology, qnap, or if I build my own. What I want to do is have 2 physical network interfaces and set up one of those interfaces to be used for torrents only.

It looks like Synology added a feature to set this up but I couldn't find a lot of information or documentation about it. Does anyone have a recommendation or tips to set something like this up?

# ¿ Jan 21, 2016 22:20

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Crashplan gets a lot of well deserved love, but does anyone use SpiderOak? They are offering unlimited data backup for $149/yr until April 22nd.

# ¿ Apr 18, 2017 19:43

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

DrDork posted:

Out of curiosity, what is it that SpiderOak does that CrashPlan does not? CrashPlan individual is $60/yr and Family is $150/yr.

I don't really know, they seem pretty comparable and I was thinking about finally getting some backup service, more for family members than myself. Spideroak seems heavily focused on privacy and the fact that you hold onto your own encryption keys. I know you can configure Crashplan like that as well though.

# ¿ Apr 19, 2017 05:01

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

The seedbox I was using offered Open VPN as well, so I setup a local Ubuntu VM that only has an internet connection that goes over that VPN and downloaded files directly to a shared drive on my local network. This makes the webUI inaccessible outside of your local network though.

You can run sonarr/radarr on the same VM or on another device if you want to make their interfaces available outside of your local network.

# ¿ Apr 29, 2017 22:19

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

DJ Burette posted:

I'm running a server at home for Plex + nzb's that I've got pretty much automated now through programs like Sonarr, Radarr, etc. It's been coming together over a number of years so has 6 disks of varying sizes and with data on, and it's running Windows 10 at the moment which I want to stick with.

I've been looking for a raid style solution for it and had originally settled on snapraid, however I've been playing with it a bit and feel like I'm swimming in a sea of unsupported powershell scripts and random forum posts from years ago as to how to make it do simple tasks like sync daily and alert me when a drive drops out. I tried using the (unofficial) Elucidate gui too but it seems to be so outdated that it couldn't even create a working config file anymore. Is there a better way out there?

I know about flexraid but I've also seen the many posts complaining about it across the internet, is it nearly as bad as people seem to make out, or would that do what I want without making me feel like I'm always guessing if it's actually properly giving me redundancy?

Check out drivepool from stablebit. It's free for 30days and just $30 for a license if you want to keep it. Adding drives to the drivepool doesn't erase the data on them which makes trying it out easy.

I set it up recently and was very impressed with how easy and functional it was.

# ¿ May 1, 2017 19:01

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

DJ Burette posted:

I'm using their scanner so I'd quite like to use drivepool too but I was under the impression that it just did drive mirroring? I'm looking for a raid6 style solution at the moment as I've got 6 full drives and only space in my case for 2 more.

Unfortunately drivepool isn't nearly as efficient with space compared to RAID6, especially as you add more hard drives. It's just duplicating files so their RAID 6 equivalent reduces your available space by 1/3 rather than just by 2 drives.

The reasons I liked it were you could set duplication on a per folder basis rather than having to use the same duplication across the entire array and that the array doesn't die if too many disks die, you always have the data on the working drives available.

I did read that is extremely annoying to restore from Crashplan after a hard drive dies with Drivepool, Crashplan doesn't have an option to leave existing files in place when restoring from a backup so you either need to download your entire backup and restore your entire pool or manually figure out which files need to be restored.

# ¿ May 1, 2017 19:58

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Thermopyle posted:

Then why have even one parity drive!?!

I'm running my home storage with no parity with this logic. I'm using stable bit's Drive Pool though so if a drive fails the entire thing doesn't go down, just what was on that disk.

This is made a lot more practical with how good a deal crash plan is, if I wasn't backing up all of my data for practically nothing off-site I would probably do raid 6 and only cloud back up important/personal things.

# ¿ Jul 19, 2017 15:03

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

eames posted:

I have a similar setup with one parity drive + n data drives. If one drive fails the parity will reconstruct it, if a second one fails during that process then only the data on the failed drives is lost. Healthy drives remain intact and JBOD readable.
All the really important data, which isn't a lot, is rsynced across all drives once a day (and of course backed up) and the probability of 5 drives dying at the same time is fairly low. Neat side effect is that you don't have to spin up all drives to read from the array.

One extra thing for anyone using or planning to use CrashPlan as a way to deal with a drive failure I should have mentioned in my original post is that Crashplan doesn't have an option to restore only missing files, so it can be difficult to figure out what specific files were stored on the failed drive and actually need to be restored.

I like drive pool a lot but was thinking about trying flexraid soon so I could get 1 parity disk and mostly avoid that issue.

# ¿ Jul 19, 2017 18:01

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

mobby_6kl posted:

Speaking of redundancy, I recall there being a system that let you set up your storage such that part of it was fully mirrored for the critical data, and the rest was lower redundancy RAID or JBOD or something. Seems like that would be the best thing for me as I could keep a small amount of important documents highly secure, and leave my Linux ISO archive exposed to somewhat higher risk. Does such a thing still exist?

If you're on windows Stablebit Drivepool lets you set duplication on a per folder basis. But duplicating across hard drives isn't really protected your files, make sure you have a real backup solution.

# ¿ Nov 5, 2017 16:38

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Drivepool is extremely cool and good, it's shocking how well it works. The main thing you have to be aware of though is that it handles file redundancy by just duplicating files. If you're using a lot of drives this is a problem because you are losing 50% of your usable space to redundancy. I remember reading posts on combining DrivePool with another piece of software to give your pool traditional RAID5-like parity based redundancy though. I'm sure you can find details for that pretty easily.

I would avoid using Windows Server for Storage though if I was setting things up again (I am just using Win10Pro atm though).
Docker for Windows works, but unreliably. I instead set up an Ubuntu server in Hyper-V, and that works a lot better, but I still run into occasional strange issues when I need to get docker/linux things to work with my Windows based network share. Windows Server supports NFS while Win10Pro does not, so you could have better luck with a proper Windows Server setup.

# ¿ Jun 20, 2018 01:43

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

I experimented with a $5 droplet to pass through traffic to a local VM, it worked pretty well.

Connected a local Ubuntu VM and the droplet with OpenVPN, setup the droplet to passthrough traffic from the WAN over the VPN to my UbuntuVM.
I tested it a bit with Emby, site was just as responsive and could do about 70mbps of bandwidth, plenty for several streams.

I had pretty ideal conditions though, gigabit FIOS to a nearby DO Datacenter, with my test user also nearby on gigabit FIOS. Also probably simpler to terminate the SSL connection at the droplet, then reverse proxy over the VPN to whatever internal devices/applications you wanted.

# ¿ Oct 9, 2018 19:18

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

nerox posted:

I understood some of these words.

Simpler, because even rereading what I wrote it looks like a mess.
I setup a $5/month cloud server with a reachable IPV4 address to just passthrough traffic it received from the internet to a local machine. The local machine wasn't reachable directly from the internet, but did have a VPN connection between it and the $5 month cloud server. Functionally this was acting as a load balancer, but not actually balancing anything.

This worked, but I think a better and easier solution for most people would be to find tutorials to do these 3 more typical things.
Setup a $5/month VPS
Setup VPN so the cloud VPS is reachable from your local devices
Setup a normal reverse proxy on the VPS (linuxserver\letsencrypt docker container would be my recommendation to do this).

Even more simpler,

quote:

Get a $5/mo VPS and forward traffic to its public IP back to your gear through a VPN tunnel.

This worked OK for me with Digital Ocean, very good connection Digital Ocean's DC, and enough CPU for the VPN tunnel.

# ¿ Oct 9, 2018 20:55

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

CopperHound posted:

I got my u-nas 810a case up and running. It turns out my use case involves a bit more that just bulk storage:

I have a lot to learn, but I think I really like unraid. I don't think I would trust it for anything mission critical and it doesn't have the performance of a striped array, but the JBOD implementation pretty much lets me throw whatever hardware I have at it. I can tell it not to split sub-directories across drives, so I should be able to recover stuff off of individual drives if I need to pull them out of the array to read. Adding a cache drive has helped write performance a lot. The SSD does not have any parity, but in my case I can tolerate that risk until the mover process has a chance to run.

e: I do hate how unraid's licensing is tied to physical USB sticks.

You can add a second cache disk and it will default to a mirrored pair.

# ¿ Aug 27, 2019 22:18

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

I use an nginx based reverse proxy and have basic auth set up in front of radarr/sonarr/other services, so in theory an attacker would not be able to exploit a vulnerability in an exposed app unless they could get past that. I'm using the linuxserver/letsencrypt container to do it so it adds TLS and fail2ban.

It's not as secure as a VPN but I think it's secure enough for what it protects, I think the bigger risk is someone managing to sneak something malicious into the docker containers which I have auto update.

# ¿ Oct 4, 2019 17:13

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

IndianaZoidberg posted:

I�m not sure if this is the right place to ask this, but here we go.

I was hoping someone could help me with an Unraid build (and I'm looking at Unraid just because it looks easier to use than some of the other options).

I would like to spend $1000 or less (not including drives) and it would be used for mass storage and hopefully as a Plex server.

I like this case (Rosewill 4U Server Chassis with 12 hot-swap drive bays)
https://www.amazon.com/gp/product/B00N9CXGSO/ref=crt_ewc_img_huc_1?ie=UTF8&psc=1&smid=ATVPDKIKX0DER
but I�m not married to the idea, or if there is a used solution I could find on eBay, I would be down for that as well.

I am a NAS noob and would love some help and hand-holding.

I built the "anniversary" build following a guide on serverbuilds.net and have been extremely happy with it. The motherboard for that one isn't available anymore but they have other builds worth a look.

NASKiller4.0: Variety of options for a NAS ranging from 2-10 plex streams, 6-15 HD capacity, $175-$600.
Lego: Larger dual CPU build with lots of expandability options, considered an "in progress" build so doesn't have as much info/options.
Hardware transcoding: Not a build exactly, but the idea here is to use an cheap small box just for plex/emby with hardware transcoding separate from your NAS.
DAS: Attach 15 bays to an existing PC/server.

# ¿ Oct 20, 2019 23:59

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Roundboy posted:

Sniping in with a stupid question

How do these compare to WD RED drives I usually buy? These would be roughly $100 less then the same thing, and I need to update my storage solution

The 8 and 10TB are going to be either actual WD Red or whites. The Whites are functionally rebranded Reds but you may need to tape over/pull the wire/use molex to deal with the 3.3v issue.

# ¿ Oct 25, 2019 18:00

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Smashing Link posted:

I'm thinking I read somewhere that newer PSUs don't require this fix, or do I have it backward?

Yes but not universally, so it's very much a your mileage may vary situation.

One discord I'm in recommends getting a SATA power cable splitter like these and just pulling out the 3.3v wire altogether (It's the one closest to the 'L'). Easy to do and only tool you need is a knife/credit card/guitar pick to wedge out the back of each sata connector. More reliable than tape, don't have to mess with your drive or power supply cables. Normal drives don't need this wire either so you can hook up whatever you want to the modified splitter.

If you want to go a step further you can also adjust the length of the modded cable to match your hard drives perfectly.

# ¿ Oct 26, 2019 17:20

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

nerox posted:

I am trying to setup letsencrypt on my server to get a reverse proxy going, which the unraid docker container needs port 80 to do, and my ISP is insistent that they do not block port 80. They are blocking port 80.

You can use just 443 if you dns validation.
Unraid itself tries to use port 80/443 so you can run into issues trying to give a container those ports directly. The workaround is to map container port 80/443 to host port 8080/8443 (arbitrary port numbers, can use whatever) and on your router port forward 80/443 to 8080/8443.

# ¿ Nov 9, 2019 20:01

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

If you have the cache setting for a share set to prefer it won't move it off nightly, it actually reverses the process so when the mover runs files on the array are moved to the cache (if there is available space).

# ¿ Dec 1, 2019 03:37

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

You can build a NAS cheaper than a similarly performing Synology. This guide gives a lot of options. https://forums.serverbuilds.net/t/guide-nas-killer-4-0-fast-quiet-power-efficient-and-flexible-starting-at-125/667

Benefits to doing this mostly are you can give yourself extra hard drives bays to allow for future expansion. If you need more plex streams the same site has a good guide on offloading plex to a cheap ~$100 prebuilt system with a modern version of intel quicksync. https://forums.serverbuilds.net/t/guide-hardware-transcoding-the-jdm-way-quicksync-and-nvenc/1408

# ¿ Dec 21, 2019 06:37

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

I think you need to disable strict port forwarding for other vpns, the default setup for that container is mostly configured for PIA which allows for port forwarding.

# ¿ Jan 17, 2020 15:34

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

There is a hidden folder on each "real" drive called "PoolPart.gibberish. Files in those PoolPart folders are in your Drivepool drive. You can save files to those PoolPart folders if you want to manually choose what physical drive stuff is on.

# ¿ Jan 18, 2020 15:28

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

If you want stuff to stay together the simplest way is to save it to your D or F drive outside of the poolpart folder rather than on the pooled Z drive.

# ¿ Jan 19, 2020 00:02

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Schilling for serverbuilds.net based builds for like the 5th time in this thread, they have a few builds that use a 4u Rosewill RSV-L4500 with a few modifications to make it actually quiet.
-Take out front fans entirely
-Reverse interior fan wall, replace fans with quieter ones
-replace back 80mm fans
-Use desktop style CPU coolers instead of typical low profile server heatsinks.
-Don't run fans full speed

I haven't tried this first hand but it supposedly is extremely quiet if not silent. You can't transplant a Dell server mobo into it, their anniversary build2 guide has various mostly supermicro boards that should all work. https://forums.serverbuilds.net/t/guide-anniversary-2-0-snafu-server-needs-a-friggin-upgrade/1075

# ¿ Jan 28, 2020 21:31

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Emby handles text based srt subtitles well, either embedded or as separate files. Other formats require it to transcode and burn in the subtitles.
This typically (and very broadly speaking) means it handles anime fine but will often transcodes bluray rips if you enable subtitles. It also supports a small handful of subtitle sites that it can search and grab external srt subs from.

# ¿ Mar 5, 2020 20:37

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Leave the parity off while you're doing the initial copying over and turn it on afterwards, parity calculations will slow down your transfer.
edit: and disable your cache for any shares you're copying to as well

THF13 fucked around with this message at 03:19 on Jun 16, 2020

# ¿ Jun 16, 2020 02:11

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

quote:

Yeah, who would want such a silly thing as parity calculations? Who needs them, what've they ever done for us, et cetera.

This advice is just for the initial copy to a new unraid server. If something goes wrong all his data is still there on the original hard drive he copied from. With Unraid you don't need to setup parity right away, you can add parity or expand it to 2 disk parity at any time.

# ¿ Jun 16, 2020 14:26

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

You could run an instance of Nextcloud with a reverse proxy out in front for access to files.

For getting to the NAS itself via smb/ssh or its management page I would still use a VPN rather than exposing either to the internet.

# ¿ Jul 15, 2020 05:41

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Has anyone in this thread ever actually lost data due to corruption that ZFS could actually recover from?
The protections while nice seem to cover problems a lot less likely than losing your array in a fire, theft, ransomware attack, software (looking at you Emby) deleting libraries, or user error.

# ¿ Sep 14, 2020 01:44

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

H110Hawk posted:

You shuck em. Aka crack open the hard exterior for their juicy disk interior. Then you throw away everything that isn't the hard drive.

Most of these will be WD whites which are basically Reds, but most whites will adhere to the SATA spec to reset the drive with 3.3v power. There's a few ways to avoid this, simplest being taping over the pins, but the better way I think is to get a 4 way sata power splitter and physically remove the wire closest to the "L" on the connector.

# ¿ Oct 29, 2020 19:55

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

SpaceInvader One who does a lot of very good unraid tutorials has a very basic guide to how parity works on Unraid here.
https://www.youtube.com/watch?v=HybwCOVDg9k

# ¿ Jan 18, 2021 14:43

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Teabag Dome Scandal posted:

I added another drive to my array in Unraid. Is there any benefit to balancing out the current data so it is more evenly spread out or does it not matter that much if everything new gets saved there?

I don't think so unless you have split levels manually configured. That setting could try to place new files on drives regardless of their available space.

# ¿ Apr 9, 2021 17:46

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

I use a nginx reverse proxy with basic auth to expose some services to the web. Basically you get a username/password prompt from nginx in your browser before you reach whatever service is behind it.

Reverse proxy means only 443/80 is exposed, so bots scanning IP:Ports only see an nginx server. It keeps your services from being indexed and added to shodan.io at least
I trust the authorization mechanisms of nginx more than whatever I'm exposing
Using the linuxserver/swag container (formerly linuxserver/letsencrypt) which includes fail2ban monitoring for basic auth logins.
That container is also nice because it handles renewing a letsencrypt cert, has most common security headers and things set up and their git page says when to update ssl, nginx, proxy, etc conf files.

I think the only thing I have exposed without basic auth is Emby just because it would break most apps.

# ¿ Jun 7, 2021 07:36

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Anecdotal, but enough overlap to mention anyways.
After my 7Pesh2 board fried my first unraid USB, I switched to a usb 2.0 thumbdrive which was recommended for stability and haven't had any issues in the 3+ years since. https://www.amazon.com/gp/product/B005FYNSUA/

# ¿ Mar 22, 2022 03:26

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Matt Zerella posted:

Has anyone with UnRAID successfully changed the port the webgui runs on?

I'd like to move it off 80/443 and use SWAG to reverse proxy but I can't find any reliable guides and see a whole lot of people saying it messes up.

This is the much more typical recommendation.
-Leave 80/443 to Unraid
-Port forward in your router 80/443 to your unraid server IP address but to different ports. I'll use 10443 and 10080 as an example.
-Configure the Swag container and map host port 10443 to container port 443, and 10080 to container port 80.

Incoming traffic from WAN to 80/443 will now go to the Swag container and work like you expect it to. If you have nat loopback setup on your router connections locally to your domain should work fine too.

# ¿ Mar 7, 2023 19:41

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

I would run it every 24 hours at like 4am, it does have a noticeable impact on the system. With 1TB I doubt it'll fill up that fast under normal usage.

# ¿ Mar 29, 2023 02:33

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

SpaceInvader One who does a lot of good videos on unraid put up a video yesterday showing a lot of what's new in 6.12 including some of the ZFS stuff.
https://www.youtube.com/watch?v=rEAfX75nReg
You can make a ZFS pool, or format array drives as ZFS but as a single disk pool only, still relying on unraid's parity drive for redundancy.

Unraid has already put out a new RC version since, and personally I am waiting at least ~2 weeks from any final release before updating. I'll probably update the SSD I use for appdata and media metadata/posters to ZFS, not sure about anything else just yet.

# ¿ May 18, 2023 20:20

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

Some of the benefits of ZFS rely on there being a pool. You aren't for example going to be able to correct bit rot errors on your single disk ZFS pool added to an unraid array. I think unraid also plans to add more ZFS features over time and not everything will be in the initial 6.12 release?

# ¿ May 18, 2023 21:21

Adbot: ADBOT LOVES YOU

# ¿ Apr 29, 2024 16:45

THF13: Sep 26, 2007; Keep an adversary in the dark about what you're capable of, and he has to assume the worst.

https://shucks.top/ keeps track of current/lowest/recent prices on the drives worth shucking. They're still a good way to get new, good drives for cheap but only when they're on sale. That happens quite frequently at least.

# ¿ May 20, 2023 04:23

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›2 »