|
Heat and noise were a major concern for me when I built out my FreeNAS box. NAS drives don't run super hot, but the extra ventilation to keep something like that cool drove me to put my server in the utility section of my basement, where the waste heat is welcome year round and the noise won't bother anyone.
|
#
?
Jun 11, 2018 21:49
|
|
|
|
| # ? May 20, 2024 01:24 |
|
|
Avian Pneumonia posted:Is there any advantage to getting an external NAS and NAS drives versus getting a few extra traditional internal hard drives and setting them up with a plex server on a raid array on my existing desktop computer? Push button configuration of your array, often push button configuration for expansion. You also suddenly get a always online target for things like Mac Time Machine backups if that's something of interest. I do just use my NAS as a file server though, it doesn't do anything else. (So: Windows, Apple, and NFS file sharing. It beacons and provides time machine backup target, I've used it for its NVR capability, music sharing, that sort of thing. Nothing "compute intensive.")
|
|
#
?
Jun 11, 2018 22:05
|
|
|
Another (related) question. I have a bunch of hard drives both internal and extrnal that i've built up over the years but i think through doing imprecise and sloppy backups that I have a lot of duplicates. Before I clean house what is the best way/program to get rid of duplicate files?
|
|
#
?
Jun 11, 2018 22:27
|
|
|
Avian Pneumonia posted:Another (related) question. If you want to try something automatic, use a tool that compares checksums between folders. YMMV, haven't used these. May be OK as a first pass but you'll probably eventually have to do some manual cleaning - eg you might have both the zip, and a copy of the files that would be inside the zip, and there's no good way to catch that automatically. I did this recently, what I did was basically "bucket sort". Sort each individual drive into a series of directories, like "/music /movies /documents /pictures" or whatever. Then merge all the various drives into one big directory, so you have one "/movies" containing all the files from all the drives. Then you can do manual deduplication more quickly in one place, instead of deduplicating five times across your five drives, or whatever. If you have different versions of the same file, it gets even more complex. There's a tool called rsync that is nice for doing this, it's capable of doing the merge and keeping the most recent version of the file, assuming that's correct. If you think you've copied an older version of a file more recently than a newer version, there's no good way to dedupe that, other than manually looking at it and picking the most recent version. Best you can do is have it keep both copies and come back and look at it (in which case, it would be nice to have had one of those automatic tools cull the outright binary duplicates so you can only have to worry about the versions that have changed). (this is also common if you've got a digital camera, you'll have like five different versions of "DCIM_1001.jpg" since that's what it defaults to after you put a new card in it, so you would not want to dedup these by date) Unfortunately there's no good automatic way to do everything, so you'll just have to sit down for a couple evenings and plow through it, just like any other mess you'd have to clean up. If you have years and years of haphazard file management and duplication, it could take a while. ZFS can be a nice safety net here since you can set snapshots and if you think you made a mistake you can easily "undo" it, but you have to have enough space to contain the complete set of everything merged together (moving/deletes are free). Paul MaudDib fucked around with this message at 03:49 on Jun 12, 2018 |
|
#
?
Jun 11, 2018 22:53
|
|
|
Avian Pneumonia posted:Another (related) question. How much data are you talking about here? If it's a hundred gigs of various copies in total just keep it all. If there are some bulky directories (music, movies) those should be excluded as they are trivially merged. Pictures are the tricky one and I just err on keeping everything.
|
|
#
?
Jun 11, 2018 23:03
|
|
|
I built a machine to set up a NAS with some disks I have lying around, so far pretty impressed with unraid, it's so incredibly easy to use. I like that you set up the shares and then don't have to care about which disks hold your data. However, I'm only getting around 60MB/s transfer speed, when gigabit ethernet is supposed to go up to 125MB/s, both machines are connected via ethernet cable to the same gigabit switch, what else could be causing it not to reach full speed?
|
|
#
?
Jun 12, 2018 00:22
|
|
|
125 sounds about right if it's still building the parity disk? If you really want to turbo charge it, slap a SSD in there and set it as a cache drive (and set your dockers to run off of it too).
|
|
#
?
Jun 12, 2018 00:24
|
|
|
Parity is built and I have an ssd as cache (although it's pretty old so might not help as much)
|
|
#
?
Jun 12, 2018 00:26
|
|
|
Simone Poodoin posted:Parity is built and I have an ssd as cache (although it's pretty old so might not help as much) Is the share set up to actually use the cache?
|
|
#
?
Jun 12, 2018 00:30
|
|
|
Simone Poodoin posted:I built a machine to set up a NAS with some disks I have lying around, so far pretty impressed with unraid, it's so incredibly easy to use. I like that you set up the shares and then don't have to care about which disks hold your data. Can you run "iperf" on both machines? Before looking too hard at your setup I would make sure your switch can actually push the packets.
|
|
#
?
Jun 12, 2018 01:01
|
|
|
H110Hawk posted:Can you run "iperf" on both machines? Before looking too hard at your setup I would make sure your switch can actually push the packets. Interesting, so judging by this the network is fine and the slowdown would be somewhere else 
|
|
#
?
Jun 12, 2018 01:18
|
|
|
Simone Poodoin posted:Interesting, so judging by this the network is fine and the slowdown would be somewhere else Correct. I would run `top -c` on your unraid machine while copying a massive file. See if any single cpu thread is pegged out. `iostat -x 2` (ignore the first output always) will show you disk subsystem stuff. Open a few terminals and get this stuff going. Is the problem when reading, writing, or both? Writing points to disk subsystem slowdowns, cpu parity calculation (unlikely), or smb implementation being poor one one side or the other. Disk subsystem could be a single slow drive dragging down your speeds.
|
|
#
?
Jun 12, 2018 01:25
|
|
|
Mystery solved, these old drives are slow as hell.  I plan on replacing them soon anyway so no worries, will deal with it in the meantime. I was planning to cannibalize my 4tb drive to add it to the array but better keep it intact in the meantime in case these die. Thanks for the help! e: nvm, the test ran while parity check was running, hence the horrible results, ran it again and got around 100MB/s for both disks Simone Poodoin fucked around with this message at 03:15 on Jun 12, 2018 |
|
#
?
Jun 12, 2018 02:13
|
|
|
Wait, you're writing directly to the disk? Did you not set the cache? Generally the way unraid works is all writes go to the SSD and the mover will move them over at night.
|
|
#
?
Jun 12, 2018 03:50
|
|
|
Right, I did set up the cache but I'm doing the initial data move and it filled up. Will check performance again when I add new disks but for now I just hook up my usb drive and copied the initial data using rsync to the user share.
|
|
#
?
Jun 12, 2018 04:07
|
|
|
Simone Poodoin posted:Right, I did set up the cache but I'm doing the initial data move and it filled up. Will check performance again when I add new disks but for now I just hook up my usb drive and copied the initial data using rsync to the user share. Ahhh gotcha. Things will definitely speed up once the mover does it's thing.
|
|
#
?
Jun 12, 2018 05:19
|
|
|
Related followup: I was considering a NAS but it seems that just geting 2 large internal hard drives in a RAID 1 array with a plex server setup is going to be best for my needs. Any internal hard drive recommendations? https://www.amazon.com/gp/product/B075WYBQXJ/ref=ox_sc_act_title_1?smid=ATVPDKIKX0DER&psc=1 This seems like a good value and should be able to stream/transcode 1080 video, yeah?
|
|
#
?
Jun 13, 2018 22:07
|
|
|
Literally any modern hard drive can read data far more than fast enough to stream 1080p video. I'm personally soured on Seagate given my 100% DOA rate on the last batch I ordered, but I still have no good explanation for why. If you don't mind shucking, WD 8TB drives are way cheaper. If you don't want to do that, Newegg has some open-box HGST for $219: https://www.newegg.com/Product/Product.aspx?Item=N82E16822146142R
|
|
#
?
Jun 13, 2018 22:33
|
|
|
I needed a little kapton tape to mask off a pin on the white label EasyStore drive to make it work with my storage array. May not be necessary for everyone but the backplane I have sure cares.
|
|
#
?
Jun 13, 2018 22:52
|
|
|
necrobobsledder posted:I needed a little kapton tape to mask off a pin on the white label EasyStore drive to make it work with my storage array. May not be necessary for everyone but the backplane I have sure cares. Was that the PWDIS pin on the power connector? Another workaround is using 4pin molex to sata power connectors cables since it only connects the +12V and leaves that pin (P3) alone.
|
|
#
?
Jun 13, 2018 23:01
|
|
|
Nope, SATA pins 5 and 6. They’re reset pins basically and they’re getting pulled up. I knew something was up when I saw the power state LED on those drives lit at full.
|
|
#
?
Jun 13, 2018 23:36
|
|
|
So I've been sitting around with 3 95% full HDs for like 2 years and we all know it's just a matter of time before one junks out and I lose a bunch of poo poo. I do have most of my favorite things backed up to a cloud service but I'd like to have something more local that I can treat like a giant HD and just FTP all my big files to, and I'd like some redundancy so it could take at least a single drive failure. Am I right in thinking my best bang-for-buck choice is to just build a tiny PC with a bunch of HD slots and run FreeNAS from a USB/small internal drive? Late last year there were some people throwing around system parts lists, are any of those still current? In my ideal scenario I'd be able to start with a pair of drives and just casually add more as I need them. I'll also most likely be running my Plex server off of it but that's currently running from my 4yr old Lubuntu desktop so my streaming needs are pretty light as it is. Would there be any issue backing stuff up from both Windows and Linux machines? Recommended/ideal RAID flavor to use? Does this still sound like a decent solution for me, and if so is there a babby's first NAS parts list anyone can point me to?
|
|
#
?
Jun 15, 2018 00:36
|
|
|
Takes No Damage posted:So I've been sitting around with 3 95% full HDs for like 2 years and we all know it's just a matter of time before one junks out and I lose a bunch of poo poo. I do have most of my favorite things backed up to a cloud service but I'd like to have something more local that I can treat like a giant HD and just FTP all my big files to, and I'd like some redundancy so it could take at least a single drive failure. What's your budget? I mean, I'd recommend a Fractal R5 case, a Supermicro Xeon-D 1541 board (a little older, but plenty for FreeNAS, CPU and mobo integrated), 16-32 GB of ECC RAM, and some WD REDs for "cheap", power efficient, and reliable. Might be able to come in around $1100. Throw in a used, small SSD for your boot drive ... USB boot has screwed me over too many times.
|
|
#
?
Jun 15, 2018 01:05
|
|
|
The R6 is less than ten bucks more. I have a R4, R5, and R6 in my house. Get the R6 if you’re going Fractal Design. Edit: Ooh, wait. The R5 has 8 2.5” trays (which is why my file server sits in one). The R6 has six, I think. Get an R5 Tapedump fucked around with this message at 01:41 on Jun 15, 2018 |
|
#
?
Jun 15, 2018 01:31
|
|
|
When I refresh my hardware on my unraid setup it’ll probably be a xeon-d, great little rigs. Will definitely get something with an ipmi port to run it truly headless (although not a huge deal with unraid anyway)
|
|
#
?
Jun 15, 2018 01:36
|
|
|
Takes No Damage posted:So I've been sitting around with 3 95% full HDs for like 2 years and we all know it's just a matter of time before one junks out and I lose a bunch of poo poo. I do have most of my favorite things backed up to a cloud service but I'd like to have something more local that I can treat like a giant HD and just FTP all my big files to, and I'd like some redundancy so it could take at least a single drive failure. I can go into more detail about parts for my build if you want. FreeNas should be stable enough with Ryzen if you're committed to that. Otherwise using Linux (openmediavault is good or any flavor really), Windows, or whatever really. Comes in around $400 and even less if you don't need more than 4 drives (5 including M.2 if u add that for system). For "raid" I really like snapraid. It doesn't stripe so even if you have drive failures beyond what is protected by parity you only lose at most the data on those drives. And you can still do drive recovery. The main downside is it is aimed at archival (which is fine for media storage overall) so you have to perform syncs to create parity and it isn't real time.
|
|
#
?
Jun 15, 2018 01:57
|
|
|
I just stuffed an R5 with 8TB EasyStores, an SSD for a cache drive and installed unRAID on it. Couldn't be happier. Runs a bunch of dockers for my media/usenet crap, VMs for programming and etc. I went Intel but I'm sure Ryzen is fine too. Stability, performance and reporting is all wonderful. My dockers all auto-update once every few weeks and I only login to the machine once or twice a month. The WebGUI they have is pretty nice too. I did a few restores to test things out, rebuilds took forever but par for the course. It's just Linux ISOs anyway so if it all goes up in smoke then whatever. I have a directory listing of them in a text file if I have get them again.
The Gunslinger fucked around with this message at 14:20 on Jun 15, 2018 |
|
#
?
Jun 15, 2018 14:18
|
|
|
Does anyone use Duplicati on a Linux NAS? How is it for resource usage for backing up a large amount of storage (20TB or so?). In particular compared to CrashPlan. I need to figure out if I should switch to CrashPlan Biz for a year or Google Enterprise and use Duplicati. If Duplicati uses significantly less resources I’d probably swap to that otherwise inertia would leave me with CrashPlan.
|
|
#
?
Jun 15, 2018 14:23
|
|
|
Hughlander posted:Does anyone use Duplicati on a Linux NAS? How is it for resource usage for backing up a large amount of storage (20TB or so?). In particular compared to CrashPlan. I need to figure out if I should switch to CrashPlan Biz for a year or Google Enterprise and use Duplicati. If Duplicati uses significantly less resources I’d probably swap to that otherwise inertia would leave me with CrashPlan. Not the same, but very similar, I use Duplicacy with its built-in encryption on my home FreeNAS array to back up about 10TB to Backblaze B2, and I couldn't be happier with it. Set it up, make the cron job, and forget about it. It just works. I really like B2, as well. Very fair pricing, and my test restores have worked properly.
|
|
#
?
Jun 15, 2018 14:32
|
|
|
Hughlander posted:Does anyone use Duplicati on a Linux NAS? How is it for resource usage for backing up a large amount of storage (20TB or so?). In particular compared to CrashPlan. I need to figure out if I should switch to CrashPlan Biz for a year or Google Enterprise and use Duplicati. If Duplicati uses significantly less resources I’d probably swap to that otherwise inertia would leave me with CrashPlan. I also use Duplicacy (not Duplicati) like the poster above me and its' resource usage is orders of magnitude less than CrashPlan.
|
|
#
?
Jun 15, 2018 16:43
|
|
|
Tapedump posted:The R6 is less than ten bucks more. Wait, I thought the R6 has space for more drive cages? Also, you mean 3.5, not 2.5? I’m trying to transition from a Lian Li A104 that has some drives shoved into the 5.25” space to make it up to 10 drives and 8 is my absolute minimum. Otherwise, I’m sticking with my rather limiting UNAS 800 case it seems.
|
|
#
?
Jun 15, 2018 18:02
|
|
|
insularis posted:What's your budget? I mean, I'd recommend a Fractal R5 case, a Supermicro Xeon-D 1541 board (a little older, but plenty for FreeNAS, CPU and mobo integrated), 16-32 GB of ECC RAM, and some WD REDs for "cheap", power efficient, and reliable. Might be able to come in around $1100. Throw in a used, small SSD for your boot drive ... USB boot has screwed me over too many times. No hard budget, this is as much a science experiment and something geeky to do as it is working up effective data backup/archiving. I'm just looking for a good value for the cost. Looks like I can get a WD Green SSD for around 40bux so that shouldn't be an issue. Looking at the Supermicro specs it only shows 6 SATA plugs, if I ever filled out the 8 HD slots in the case how would I plug them all in? Speaking of, my PC now is already out of SATA power and I'm wary about using one of those little molex -> SATA converters because one shorted out and let out the magic blue smoke on me a few years ago. Is there a recommendation for power supplies that have that many built in SATA plugs?
|
|
#
?
Jun 15, 2018 19:06
|
|
|
Takes No Damage posted:No hard budget, this is as much a science experiment and something geeky to do as it is working up effective data backup/archiving. I'm just looking for a good value for the cost. Looks like I can get a WD Green SSD for around 40bux so that shouldn't be an issue. The usual way is a used IT mode HBA like the LSI1068 (I think that's the one) or any basic LSI HBA plus a couple of 8087 cables that go from SAS to 4 SATA data lines. eBay this part, you can get them for $40. As for power connectors, get an Antec or Seasonic power supply with the modular plugs (they provide a bunch in the package), and you can use as much SATA power as needed.
|
|
#
?
Jun 15, 2018 19:51
|
|
|
Anyone have any real world experience with ZFS record sizes? I'm considering moving my media folders to a 1MB record size, but at 20TB of moving stuff around for it to take effect on existing files, I thought I'd ask if it made much difference before rsyncing for two days.
|
|
#
?
Jun 15, 2018 19:54
|
|
|
necrobobsledder posted:Wait, I thought the R6 has space for more drive cages? Also, you mean 3.5, not 2.5? It comes with six trays but has slots for at least two more midway, and I think at least one more near the top. 
|
|
#
?
Jun 15, 2018 20:26
|
|
|
Takes No Damage posted:So I've been sitting around with 3 95% full HDs for like 2 years and we all know it's just a matter of time before one junks out and I lose a bunch of poo poo. I do have most of my favorite things backed up to a cloud service but I'd like to have something more local that I can treat like a giant HD and just FTP all my big files to, and I'd like some redundancy so it could take at least a single drive failure. If you're talking about bang for buck, here's your semi-comedy ultra budget refurb option. CPU: Xeon X3440 - $15 CPU cooler: Intel stock - $9 Motherboard: Supermicro X8SIL-F - $40 (the non-F version is cheaper but this has out of band management) RAM: 2x 4GB Hynix DDR3-1866 ECC UDIMM - $44 (will need to run at 1333 but that's OK, this stuff has timings all the way down to 800) Case: Fractal Design Node 804 - $106 PSU: whatever you feel like paying for Total: $214 before PSU and drives Obviously you could get a cheaper case and go even lower, but how many non-gargantuan cases come with 10 3.5" bays? Even if you need to add a SATA/SAS controller to use them all, you're set for a few years down the road when your first RAID fills up and you have to add a second one. Eletriarnation fucked around with this message at 21:32 on Jun 15, 2018 |
|
#
?
Jun 15, 2018 21:29
|
|
|
Even better option for the motherboard - X8SI6. Has an onboard LSI 2008 that flashes to IT mode just fine.
|
|
#
?
Jun 15, 2018 22:16
|
|
|
^^^ So with this MOBO I wouldn't need a SAS PCI card? e2: This review makes it sound even better: "For all intents and purposes, the Supermicro X8SI6-F’s onboard LSI SAS 2008 controller is equivalent to the LSI 9211-8i add-in card. Just for reference, the LSI 9211-8i retails for around $260 US. The important thing here is that even if the onboard controller did fail, one could purchase a compatible add-in card and recover the array. Just to test this, using Windows Server 2008 R2 I was able to unplug drives (all idling) from the onboard controller and into a LSI 9211-8i controller without the OS and was able to access the drives seconds later." insularis posted:The usual way is a used IT mode HBA like the LSI1068 (I think that's the one) or any basic LSI HBA plus a couple of 8087 cables that go from SAS to 4 SATA data lines. eBay this part, you can get them for $40. Oh yeah, I forgot PCI cards were a thing  Looks like Amazon has those breakout cables for twelve bucks, is that what you were talking about?insularis posted:As for power connectors, get an Antec or Seasonic power supply with the modular plugs (they provide a bunch in the package), and you can use as much SATA power as needed. My last 2 PSUs were some Seasonic modular ones and I'm still out of SATA power  I even tried to order another plug for additional SATA power cords from ModDIY, but USPS lost the package... Of course I am running 2 optical drives in addition to a pair of both SSDs and big platter HDs so I guess that is a lot, and with a dedicated storage server those could all go to HDs.Eletriarnation posted:If you're talking about bang for buck, here's your semi-comedy ultra budget refurb option. Hey thanks for all this. If I mash that together with previous recommendations I'm sure I can cobble together something functional that I can use for at least a few years. Takes No Damage fucked around with this message at 22:40 on Jun 15, 2018 |
|
#
?
Jun 15, 2018 22:20
|
|
|
Oh and because I just noticed the LSI 1068 mentioned - don't get that card. Those early-gen SAS cards can't handle >2TB drives.
|
|
#
?
Jun 15, 2018 22:22
|
|
|
|
| # ? May 20, 2024 01:24 |
|
|
IOwnCalculus posted:Even better option for the motherboard - X8SI6. Has an onboard LSI 2008 that flashes to IT mode just fine. I found these instructions for flashing to IT on a PCI card, do you know if the same steps would work for the onboard from Supermicro X8SI6-F or is that a different process?
|
|
#
?
Jun 15, 2018 22:52
|
|