|
Combat Pretzel posted:The USB stick possibly not respecting flushes, potentially causing more data loss than necessary on power loss. ZFS uses flushes to force harddisk and SSD controllers to write things in a certain order. Then again, this is (or was) also an issue with cheaper harddisks. To further elaborate, what I am envisioning is a laptop and a handful of 2.5" USB hard drives, probably in RAIDZ-2. Frequent power loss is probable, so everything will have backup power (the laptop inherently plus something I will probably have to build for the hard drives*). So far this is the best I've come up with that meets the requirements (very small/portable, serviceable, ability to use laptop as a normal computer with the array and disks unmounted). *Does anyone know of a <=15" laptop with 5+ USB 3.0 ports? 
|
#
?
Oct 22, 2015 11:56
|
|
|
|
| # ? May 29, 2024 21:17 |
|
|
sleepy gary posted:To further elaborate, what I am envisioning is a laptop and a handful of 2.5" USB hard drives, probably in RAIDZ-2. Frequent power loss is probable, so everything will have backup power (the laptop inherently plus something I will probably have to build for the hard drives*). So far this is the best I've come up with that meets the requirements (very small/portable, serviceable, ability to use laptop as a normal computer with the array and disks unmounted).  This sort of setup will probably only cause you pain. Look at a small NAS that can do iSCSI and use an Ethernet crossover cable and save some sanity.
|
|
#
?
Oct 22, 2015 14:25
|
|
|
Buy a laptop when Thunderbolt 3 is a thing and then use an external array.
|
|
#
?
Oct 22, 2015 19:31
|
|
|
mayodreams posted:
Do you have any specific reasoning? A small NAS like a DS414slim is $300 without disks which would increase my storage costs by like 70%. In spite of being a very nice size on its own, it significantly complicates the matter of backup power which in turn reduces portability by a lot (would need a full-fledged inverter-based UPS and its associated cost and weight). It would also add another point of failure, because it's another device that can die on me in addition to the laptop and the disks and it would be harder to replace on short notice. Unless there is something I am overlooking, I don't think all those downsides are worth creating a slightly less janky setup.
|
|
#
?
Oct 22, 2015 21:33
|
|
|
sleepy gary posted:Do you have any specific reasoning? Why do you need ultra-portability in your NAS?
|
|
#
?
Oct 22, 2015 21:36
|
|
|
I know a guy that was doing a lot of recording projects for a studio that kept all his work with him as he traveled around and went with a 4-bay NAS with an eSATA port in the end.
|
|
#
?
Oct 23, 2015 00:04
|
|
|
sleepy gary posted:To further elaborate, what I am envisioning is a laptop and a handful of 2.5" USB hard drives, probably in RAIDZ-2. Frequent power loss is probable, so everything will have backup power (the laptop inherently plus something I will probably have to build for the hard drives*). So far this is the best I've come up with that meets the requirements (very small/portable, serviceable, ability to use laptop as a normal computer with the array and disks unmounted). Jesus christ go buy yourself a Drobo Mini already. It's exactly what you want, right down to battery backup for data in its internal RAID controller RAM. A little expensive but it will actually work, and in a single small box instead of needing a rats nest of USB cables and poo poo. It has four drive bays and there are 2TB 9.5mm drives out there so max capacity is 6TB with single redundancy.
|
|
#
?
Oct 23, 2015 02:27
|
|
|
He sounds pretty dead set on his mobile cyberpunk fantasy, I doubt he can see the catastrophe looming when he plugs them in in the wrong order, or god knows what else will come up.
|
|
#
?
Oct 23, 2015 02:49
|
|
|
Not sure why you guys are getting hostile. I am asking you for pitfalls I haven't considered but nobody has given me any other than what I already knew, which is that it will be a little unwieldy when everything is attached. You can't plug drives in in the wrong order for ZFS arrays. The Drobo mini looks pretty cool but it's $570 plus the cost of an msata drive for the battery backup system to work, and I can't swing that. So far what you guys are saying is that I should spend $300-$570+ so I don't have to deal with (as many) USB cables. edit: Here is what 4 USB hard drives attached to a laptop looks like (not my photo):  That's not some nightmare rat's nest of cables worth spending $300+ to avoid without some other really good reasons. sleepy gary fucked around with this message at 04:22 on Oct 23, 2015 |
|
#
?
Oct 23, 2015 04:01
|
|
|
sleepy gary posted:Not sure why you guys are getting hostile. I am asking you for pitfalls I haven't considered but nobody has given me any other than what I already knew, which is that it will be a little unwieldy when everything is attached. You can't plug drives in in the wrong order for ZFS arrays. How important IS this data to you? Is it going to be backed up offsite? What is the main use case for it being so portable? You have a pretty good chance of mucking something up doing it the way you want.
|
|
#
?
Oct 23, 2015 04:19
|
|
|
Skandranon posted:How important IS this data to you? Is it going to be backed up offsite? What is the main use case for it being so portable? You have a pretty good chance of mucking something up doing it the way you want. It's very important and will be backed up onsite and offsite, but offsite might not keep up with how quickly data is generated for some periods. The use case for it being so portable is the need to travel around with it and a shitload of other stuff. Would you please tell me actual reasons why I have a good chance of mucking something up doing it the way I want? That's what I'm asking in the first place.
|
|
#
?
Oct 23, 2015 04:27
|
|
|
sleepy gary posted:It's very important and will be backed up onsite and offsite, but offsite might not keep up with how quickly data is generated for some periods. The use case for it being so portable is the need to travel around with it and a shitload of other stuff. Would you please tell me actual reasons why I have a good chance of mucking something up doing it the way I want? That's what I'm asking in the first place. Just the fact you have them loose leads to a big increase in the chance the drives fail, due to travel, dropping, or some other unforseen event. Is there a reason you want 5x ZFS instead of just RAID-1?
|
|
#
?
Oct 23, 2015 04:39
|
|
|
sleepy gary posted:Not sure why you guys are getting hostile. I am asking you for pitfalls I haven't considered but nobody has given me any other than what I already knew, which is that it will be a little unwieldy when everything is attached. You can't plug drives in in the wrong order for ZFS arrays. 
|
|
#
?
Oct 23, 2015 04:44
|
|
|
Skandranon posted:Just the fact you have them loose leads to a big increase in the chance the drives fail, due to travel, dropping, or some other unforseen event. Is there a reason you want 5x ZFS instead of just RAID-1? Being inside some RAID or NAS enclosure doesn't give hard drives any extra protection from falls though so where is the benefit? Also, it's not like the drives are just going to be knocking into each other in a bag or something, they will be in some decently padded casing. I am preparing for hardware failure by using parity and backups. I am further preparing for hardware failure by trying to stick to standard off the shelf stuff. As I said before, if a laptop or a hard drive dies, you can replace that same-day, almost anywhere in the world. You probably can't say the same for specialty micro-NAS appliances. Cost is also a problem... I don't have a lot of money and I'm trying to make the best solution I can with the financial and physical constraints. RAID-1 would limit me to the capacity of a single disk (assuming one array) which would not work with 2.5" drives. With 3.5" drives it might be fine, but then I think I am getting into space, weight, and power issues again.
|
|
#
?
Oct 23, 2015 04:56
|
|
|
sleepy gary posted:Being inside some RAID or NAS enclosure doesn't give hard drives any extra protection from falls though so where is the benefit? Also, it's not like the drives are just going to be knocking into each other in a bag or something, they will be in some decently padded casing. I am preparing for hardware failure by using parity and backups. I am further preparing for hardware failure by trying to stick to standard off the shelf stuff. As I said before, if a laptop or a hard drive dies, you can replace that same-day, almost anywhere in the world. You probably can't say the same for specialty micro-NAS appliances. Cost is also a problem... I don't have a lot of money and I'm trying to make the best solution I can with the financial and physical constraints. They are both vulnerable to damage the same way IF they fall, but you having them loose gives you so many more opportunities to make a mistake with them. If you are seriously cost constrained and need to be able to carry this much data, then do what you must, but there are risks to it. You are trying to perform a task that is inherently costly, and trying to cut back on the cost. Also, any NAS appliances that has been suggested would take the same hard drives you are planning to use.
|
|
#
?
Oct 23, 2015 05:01
|
|
|
Alright thanks for your input everyone. No need for anything else on this topic.
|
|
#
?
Oct 23, 2015 05:04
|
|
|
I don't think the drobo mini needs the mSATA drive installed to have its battery backup work, it's for making the thing faster by essentially caching some of the data on the SSD. It sounded completely optional when I read up about it before (never bought one though because I don't really need it). To me the odd thing is the combination of all these requirements you've got (frequent travel => highly portable, advanced RAID for high reliability, battery backup for even higher reliability) with near-zero budget. Sure, from one perspective $600ish is a lot for an empty container for four 2.5 inch drives, but that should be peanuts compared to the costs of routine business travel and it's only a one-time cost. And you were talking (joking?) about buying an entire laptop just to get one with a shitload of USB3 ports built in, which would be a few hundred at least (and probably more than $600 if you bought anything of decent quality which would stand up to lots of field use). If you have a need to collect terabytes of data and you're paranoid enough about losing any of it to invest in RAID at all (as opposed to just buying a handful of 2TB or Seagate's new 4TB 2.5" USB drives and using them solo), it's really weird that you're being so cheap. Feels very penny wise, pound foolish. Oh, and my objection to the medusa setup isn't that it can't work, it's that it's lots more cables to pack/lose, many more connectors to plug/unplug/have break at inopportune times, very clumsy, and so on. Just one box external to your laptop is bad enough, but that is basically asking to have random annoyances and/or failures pop up. Also the medusa might not let you pull off your laptop battery based backup power idea. Notice how that picture of the laptop with 4 drives has a power brick connected to that USB hub? (Power brick is out of frame, but you can see the cord.) Most laptops probably won't be able to supply enough current to their USB ports to run four or more external 2.5" HDDs. e: f, b on a bunch of this oh well
|
|
#
?
Oct 23, 2015 05:07
|
|
|
BobHoward posted:To me the odd thing is the combination of all these requirements you've got (frequent travel => highly portable, advanced RAID for high reliability, battery backup for even higher reliability) with near-zero budget. Sure, from one perspective $600ish is a lot for an empty container for four 2.5 inch drives, but that should be peanuts compared to the costs of routine business travel and it's only a one-time cost. And you were talking (joking?) about buying an entire laptop just to get one with a shitload of USB3 ports built in, which would be a few hundred at least (and probably more than $600 if you bought anything of decent quality which would stand up to lots of field use). It was a joke about the laptop. It would really simplify things for me by eliminating any need for backup power for the drives, but it's not realistic so I will just make something myself to deal with that. I'm not sure where you get the idea that the budget is near-zero. It's just already way maxed out already so just casually tossing in a NAS is not trivial. quote:If you have a need to collect terabytes of data and you're paranoid enough about losing any of it to invest in RAID at all (as opposed to just buying a handful of 2TB or Seagate's new 4TB 2.5" USB drives and using them solo), it's really weird that you're being so cheap. Feels very penny wise, pound foolish. I don't follow the logic here. Using a pile of solo drives makes no sense and I don't think that having two parity disks plus a backup strategy is being "so cheap." Just because I want to avoid spending money on a NAS appliance? quote:Oh, and my objection to the medusa setup isn't that it can't work, it's that it's lots more cables to pack/lose, many more connectors to plug/unplug/have break at inopportune times, very clumsy, and so on. Just one box external to your laptop is bad enough, but that is basically asking to have random annoyances and/or failures pop up. There are lots of solutions to problems like these. I am fully aware of the power issues and I am capable of making a battery powered USB hub, and put that along with the drives into a nice case with a single USB cable coming out for connecting to the laptop. edit: It's a dead horse at this point though. I appreciate everyone's input but I don't need anything else at the moment.
|
|
#
?
Oct 23, 2015 05:32
|
|
|
sleepy gary posted:Not sure why you guys are getting hostile. I am asking you for pitfalls I haven't considered but nobody has given me any other than what I already knew, which is that it will be a little unwieldy when everything is attached. You can't plug drives in in the wrong order for ZFS arrays. Responses to your questions are less hostility, and more warnings that what you are planning is really stupid and will fail spectacularly at some at point, and take some or all of your data with it. You come to a forum with experts and ask questions and then refute the answers. But what do I know? I am just the technical lead for a prominent consumer/corporate order company that manages hundreds of terabytes of enterprise grade storage. Go right ahead with your plan!
|
|
#
?
Oct 23, 2015 13:33
|
|
|
It's the classic goon-in-the-well scenario.
|
|
#
?
Oct 23, 2015 15:12
|
|
|
I'm currently running Crashplan on a Synology DS214, and I'm using up 90%+ of RAM constantly. Would upgrading to a DS415+ (2GB RAM, more HD bays) help?
|
|
#
?
Oct 23, 2015 15:51
|
|
|
Java certainly can eat some resources.
|
|
#
?
Oct 23, 2015 16:03
|
|
|
Would this: http://www.amazon.ca/Buffalo-Tb-Sec...=3tb+hard+drive quote:Buffalo 3 Tb 4k Sector Optional Hard Work in a PC if I threw away the enclosure part and put it in a regular windows computer? Its the best price online at the moment for 3tb drives that I can find in canada. I'm wondering if there is any special firmware in it that would prevent this bad idea from taking shape.
|
|
#
?
Oct 27, 2015 21:13
|
|
|
shadow puppet of a posted:Would this: Probably, doesn't even look like it has a special backplane, it's just the hard drive sticking out the other end. I'd be wary though, you don't actually know what hard drive it is.
|
|
#
?
Oct 27, 2015 21:24
|
|
|
Skandranon posted:I'd be wary though, you don't actually know what hard drive it is. This, spend a little more and get something with good reviews.
|
|
#
?
Oct 27, 2015 21:26
|
|
|
I do not think I have seen a 4k spindle speed.
|
|
#
?
Oct 27, 2015 21:55
|
|
|
I've got an existing three disk zfs raidz1 pool. I want to mirror this pool with three new disks. What would be the magical incantation of commands to do this? Thanks. EDIT: I guess what I want to do is use the 'attach' command? Actually it looks like I can't use attach. The MAN says it can't be used on a raidz setup. Bleh. How can I mirror the existing pool? Is this really not possible without destroying the pool? xgalaxy fucked around with this message at 00:12 on Oct 29, 2015 |
|
#
?
Oct 28, 2015 23:14
|
|
|
Hmm, so are there any Xeon-D boards out yet that are rocking say 2 x8 PCIe slots for my HBA needs? That's what I'm waiting for so I can upgrade my NAS. ASRock board still vaporware?
movax fucked around with this message at 01:23 on Oct 29, 2015 |
|
#
?
Oct 29, 2015 01:19
|
|
|
movax posted:Hmm, so are there any Xeon-D boards out yet that are rocking say 2 x8 PCIe slots for my HBA needs? That's what I'm waiting for so I can upgrade my NAS. ASRock board still vaporware? We have a Supermicro 5018D-FN4T Xeon-D for PCIe interop testing at work and it has a single x16. It's a really great little box/mobo, I like it a lot. Would make a nice NAS controller with 6 onboard sata3 ports too. Quad GbE too! http://www.supermicro.com/products/system/1u/5018/sys-5018d-fn4t.cfm
|
|
#
?
Oct 29, 2015 03:49
|
|
|
movax posted:Hmm, so are there any Xeon-D boards out yet that are rocking say 2 x8 PCIe slots for my HBA needs? That's what I'm waiting for so I can upgrade my NAS. ASRock board still vaporware? Not sure what you mean by Asrock being vaporware - they had some early mini-ITX Avaton-based boards with Marvell controllers that would reportedly drop disks under heavy load, but to my knowledge those problems have been fixed (and wouldn't matter if you were using HBA(s) anyway). Unfortunately Supermicro hasn't done a micro-ATX Xeon-D board yet, as that's what I'm waiting for - especially if they feature a LSI controller on-board. BlankSystemDaemon fucked around with this message at 13:11 on Oct 29, 2015 |
|
#
?
Oct 29, 2015 13:03
|
|
|
xgalaxy posted:I've got an existing three disk zfs raidz1 pool. I want to mirror this pool with three new disks. What would be the magical incantation of commands to do this? RAID-Zn and mirroring are fundamentally different approaches to write distribution, so no, you can't convert a RAIDZ1 vdev to a mirror without going through the process of backup, recreate, and restore.
|
|
#
?
Oct 29, 2015 18:28
|
|
|
That said, why are you looking to do this? If you just want another local copy in case of hardware failure and discs in the same computer are acceptable you could add another raid z1 and do scheduled zfs sends.
|
|
#
?
Oct 29, 2015 18:51
|
|
|
Well I already have the three new drives on order. I think what I'll do is just destroy the pool and switch to three vdev's with mirrors setup. Something like this: code:
|
|
#
?
Oct 29, 2015 19:29
|
|
|
K, so. Bunch of questions for y’all. My family has two desktops, two mac books, and a laptop, totaling about 5TB. I want to set up a house server / backup / media storage center. In going through the posts, my current idea is to follow Lowen SoDium posted:post (Ubuntu, raid-z, ZFS, snapshots, with crashplan) The quoted post is ~2 years old, is anything out of date or need updating? My budget is ideally under four figures. The idea is being able to move things to the share drive, watch movies from the share drive (through kodi on my desktop), and have regular backups. - Other questions, as I've never done this before: When setting it up, do I need to use a temporary screen/keyboard? When it’s set up, can the client computers click and drag folders to the share drive? Can multiple computers stream from it at once? When it’s all connected, can I give the server commands from any of the clients? Do I need to set up a shell to do it? If my laptop is on campus, can it still access the drive? (I think I need to get a dynamic DNS service for this?) Are there any problems if I use veracrypt/filevault on the clients and then veracrypt again on the share drive? (If Im doing that and VPN to crash plan, will I be able to read files if I need to recover them?) Going a bit abroad at this point, is there a way to make the share drive the gateway port? IE, I only need a VPN for it, because all the other computers go through the internet through it. (I think its port forwarding to the share drive, but I’m not sure)
|
|
#
?
Oct 31, 2015 19:02
|
|
|
AsrockRack predictably launched some new Xeon-D motherboards, and I have to say I'm thorougly impressed by the road they went with, with the uATX series of motherboards that have can have 8 disks on (mini-)SAS3 via LSI 3008 and 6 disks via SATA3 on Intel PCH, in addition to supporting up to 128GB memory and featuring a CPU that, in addition to being passively cooled, goes almost toe-to-toe with the Intel Xeon E3-1220 v3 which has double the TDP. Add to that proper IPMI and 10GbE (or 10Gb SFP+, on some models), and it's a pretty wonderful offering. Only thing we don't know is pricing and availability, which are of couse the most relevant. There's also a M.2 via Marvell controller, if you hate yourself.
|
|
#
?
Oct 31, 2015 23:35
|
|
|
D. Ebdrup posted:AsrockRack predictably launched some new Xeon-D motherboards, and I have to say I'm thorougly impressed by the road they went with, with the uATX series of motherboards that have can have 8 disks on (mini-)SAS3 via LSI 3008 and 6 disks via SATA3 on Intel PCH, in addition to supporting up to 128GB memory and featuring a CPU that, in addition to being passively cooled, goes almost toe-to-toe with the Intel Xeon E3-1220 v3 which has double the TDP. Add to that proper IPMI and 10GbE (or 10Gb SFP+, on some models), and it's a pretty wonderful offering. Only thing we don't know is pricing and availability, which are of couse the most relevant. Given the rather ridiculous cost most of the Xeon D platforms seem to be at right now, and the general poor availability, I'm not convinced. However, if you need 10gbaseT, or 64+ gb of ram, I can see its appeal.
|
|
#
?
Nov 1, 2015 01:37
|
|
|
I've had the realisation that, since I plan to leave my file server off most of the time and only WakeOnLAN it as necessary - meaning idle power consumption is not really an issue - it might not make much economic sense to build a separate machine from my desktop... hardware-wise, that is. Software-wise, of course, is probably another matter, since the desktop must run Windows. Hypothetically, what would be the least terrible option available for using a Windows machine as a file server/torrent box/streaming host? Is ZFS off the table or are there drivers for it? Would it be as silly as it sounds to run FreeNAS in a VM under Windows?
|
|
#
?
Nov 2, 2015 13:52
|
|
|
NihilCredo posted:I've had the realisation that, since I plan to leave my file server off most of the time and only WakeOnLAN it as necessary - meaning idle power consumption is not really an issue - it might not make much economic sense to build a separate machine from my desktop... hardware-wise, that is. You really don't want to run FreeNAS in a VM unless you're doing hardware passthrough of a drive controller or drives themselves. FreeNAS (ZFS in particular) wants full control of the disks. Also virtual disk performance is awful on top of that and there could be serious data errors (so I've heard. I've run FreeNAS as a test in ESXi and the disk write performance was very poor). If your dataset isn't changing much, IE large static media collection, you could look into something like Snapraid. Just send the torrents to a scratch disk before they get loaded to the media collection. There's also Flexraid, but I don't know much about it and I believe there's a cost associated with it. Have you considered a NAS appliance like a Synology or Q-Nap? I've used both and they're great devices. They're also very low power for the most part and compact so they don't take up much space. Using your desktop as a server has some drawbacks, I used to do it years ago and I ran into all sorts of limitations and bugs. They've probably been long fixed by now, for instance running out of memory for file shares under Windows 7. That one required a registry hack to fix. Also if you're wanting a local backup of your data, a NAS is handy. (but remember to have an offsite solution too) Keeping your local backups on the same machine is asking for trouble, say the power supply goes tits up and takes all the hard drives with it. There goes your backups. Also a NAS appliance like Synology comes with great tech support. Anyway, just some jumbled thoughts.
|
|
#
?
Nov 2, 2015 15:02
|
|
|
LowellDND posted:K, so. Bunch of questions for y’all. Based on discussion with Lowen, this is what I got http://pcpartpicker.com/p/vDTxzy Loel fucked around with this message at 18:52 on Nov 2, 2015 |
|
#
?
Nov 2, 2015 18:36
|
|
|
|
| # ? May 29, 2024 21:17 |
|
|
Well, if you run Windows 8 or upwards, the Pro versions come with HyperV, which supports RAW disk access. However, no information whether it also passes flush commands from the VM to the disk. That's all ZFS cares about, that flush commands get to the drive and are actually processed (some cheaper IDE and SATA drives in the past did not).
|
|
#
?
Nov 2, 2015 20:36
|
|
