|
Saukkis posted:Sounds like instead of spending money to move the drives to you garage you should buy a pair of cheaper 500GB SSDs to replace the RAID6 array. I'd love to, but all the drives are recycled, didn't spend a dime. If I had the money, I'd replace all the RAID6 with SSDs like you said. Paul MaudDib posted:Can I use regular SATA cables between a SAS device and a SAS controller? Which SAS controller? Chances are: No, unless you have an adapter. Some SAS controllers do break out into SATA style connectors, but generally are still carrying extra pins for the SAS addressing. I'd suspect not. SAS to SATA? Probably. SAS to SAS? Likely not. You'll be missing key pins. CommieGIR fucked around with this message at 13:32 on Aug 8, 2017 |
#
?
Aug 8, 2017 13:28
|
|
|
|
| # ? May 12, 2024 17:08 |
|
|
CommieGIR posted:I'd love to, but all the drives are recycled, didn't spend a dime. If I had the money, I'd replace all the RAID6 with SSDs like you said. I meant that if you want to put the drives in the garage you will have to buy new hardware, so it's better to buy SSDs instead.
|
|
#
?
Aug 8, 2017 13:47
|
|
|
Wistful of Dollars posted:Since Google Drive has been giving my wife guff lately, I've been musing going the home NAS route as a replacement for her. All she uses the cloud storage for is photos and videos she takes while traveling, so it's not exactly taxing. The main issue is making sure files/folders on there can be remotely accessed/shared with family. Depends. Are you the kind of person who will, upon seeing the capabilities of a new device, want to maximize said capabilities beyond the device's ability? If not, grab whatever consumer storage device can handle your need at a price point you can afford. If so, you have two options: If you're just looking for file storage and nothing else, there's plenty of consumer products available that do that and nothing much else. If you want to experiment with a home NAS that can store files and eventually do other things and you really can't control yourself, then building one from scratch would be advisable. Devices from QNAP and Synology, for someone who just continues to tinker and expand your use-case, are gateway drugs that will leave you hanging. They're often powerful enough to give you glimpses of what's possible but not powerful enough to fulfill your desires.
|
|
#
?
Aug 8, 2017 13:57
|
|
|
Saukkis posted:I meant that if you want to put the drives in the garage you will have to buy new hardware, so it's better to buy SSDs instead. I think you have me mixed up. I'm not putting drives in my garage.
|
|
#
?
Aug 8, 2017 14:10
|
|
|
DrDork posted:If it's only one server to one client, Infiniband/FC is gonna end up being much cheaper than 10-GigE. It is just one client to one array. Can you get 100 foot Infiniband cables? oh around $500-600 bux. 10G ethernet might be cheaper actually. redeyes fucked around with this message at 14:33 on Aug 8, 2017 |
|
#
?
Aug 8, 2017 14:30
|
|
|
You could do 8Gb / 16Gb fiber channel if you have the right enclosure
|
|
#
?
Aug 8, 2017 14:55
|
|
|
Thanks Ants posted:You could do 8Gb / 16Gb fiber channel if you have the right enclosure I don't have anything yet. What kind of enclosure would this be? I have a rack in my garage and wanted to centralize all my junk. I really only need 150-200MB/s at this point, of course more is nice. [edit] I am not good at enterprise stuff because I work totally in small business. redeyes fucked around with this message at 15:00 on Aug 8, 2017 |
|
#
?
Aug 8, 2017 14:57
|
|
|
redeyes posted:I don't have anything yet. What kind of enclosure would this be? I have a rack in my garage and wanted to centralize all my junk. I really only need 150-200MB/s at this point, of course more is nice. Total or per device? If total, consider just using multipathed iscsi
|
|
#
?
Aug 8, 2017 15:30
|
|
|
evol262 posted:Total or per device? If total, consider just using multipathed iscsi No just total bandwidth. I have 7x Hitachi 7200RPM NAS drives. They can hit between 150-200MB/s in general for a single drive and that is just fine. I think for that to work I would need 10G ethernet?
|
|
#
?
Aug 8, 2017 15:48
|
|
|
redeyes posted:No just total bandwidth. I have 7x Hitachi 7200RPM NAS drives. They can hit between 150-200MB/s in general for a single drive and that is just fine. I think for that to work I would need 10G ethernet? How many devices do you plan to feed connections to for that? I mean, unless you are building a SERIOUS VM lab with 25-50 machines on it, just Gigabit will probably be fine, if you even saturate that. I second the call for iSCSI.
|
|
#
?
Aug 8, 2017 15:56
|
|
|
redeyes posted:No just total bandwidth. I have 7x Hitachi 7200RPM NAS drives. They can hit between 150-200MB/s in general for a single drive and that is just fine. I think for that to work I would need 10G ethernet? 10g would be ~1050 mb/s, depending on your efficiency, so no, not for all of them. Total, multipathed iscsi over gige will get you 150-200mb/s easily
|
|
#
?
Aug 8, 2017 16:52
|
|
|
CommieGIR posted:Which SAS controller? Chances are: No, unless you have an adapter. Some SAS controllers do break out into SATA style connectors, but generally are still carrying extra pins for the SAS addressing. LSI something, and I can confirm it's pinned out into four SATA-style connectors (right now I just use it as a SATA controller). This will be for an internal Quantum half-height LTO5 tape drive, p/n 12X5250. I dug up the datasheet for the modern version, which appears to have a SFF8482 connector. Here's a listing for the same item and it looks like that SFF8482 connector. OK then, so it looks like I need one of these? Or is this just a "2-in-1" connector and I could get away with separate SATA+power? Thanks for your help. I'm still finding my way around this whole enterprise storage thing. I always tossed around the idea of getting a tape drive but it wasn't worth it at $1000+. At $185 shipped I couldn't say no. Now I can have real off-site backups (tapes in my climate-controlled storage unit).
|
|
#
?
Aug 8, 2017 18:11
|
|
|
Thwomp posted:Devices from QNAP and Synology, for someone who just continues to tinker and expand your use-case, are gateway drugs that will leave you hanging. They're often powerful enough to give you glimpses of what's possible but not powerful enough to fulfill your desires.
|
|
#
?
Aug 8, 2017 18:12
|
|
|
Paul MaudDib posted:LSI something, and I can confirm it's pinned out into four SATA-style connectors (right now I just use it as a SATA controller). If the back of the drive has the combined SAS+power cable, then you'll most likely need one of https://www.pc-pitstop.com/sas_cables_adapters/sff-8087/ these guys.
|
|
#
?
Aug 8, 2017 18:42
|
|
|
beepsandboops posted:As somebody who just bought a 2-bay QNAP, are you reading my mind? As someone who has owned a 2-bay QNAP for two years now, probably. They're great little devices and if they suit your use cases and you don't ever see yourself asking more of them, they are perfect. But most people on this forum and browsing this thread are probably the tinkering type and will likely want to do more once they get a taste of what's possible. I'm just going to cross-post this here since I am looking at maybe replacing my QNAP with a home-built rig. quote:Just a quick check of something I've been putting together. I'm just looking to see what kind of budget home server I could create in the smallest form factor possible.
|
|
#
?
Aug 8, 2017 19:49
|
|
|
Thwomp posted:As someone who has owned a 2-bay QNAP for two years now, probably. Don't skimp on the memory. 8GB is not a lot these days.
|
|
#
?
Aug 8, 2017 19:56
|
|
|
Thwomp posted:
I'm using the 6100 in my server and it's great. I'm using a Gigabyte X150M-ECC motherboard, with ECC RAM. https://www.amazon.com/Gigabyte-E3-...ds=gigabyte+ecc It has no graphics output, so you need to put a GPU in whilst you install the OS for the first time, then turn off, pull the GPU and run it headless after that. But you get the advantage of a cheap ECC board for your trouble, should you need ECC. e: found a more appropriate link. apropos man fucked around with this message at 23:58 on Aug 8, 2017 |
|
#
?
Aug 8, 2017 23:56
|
|
|
Paul MaudDib posted:LSI something, and I can confirm it's pinned out into four SATA-style connectors (right now I just use it as a SATA controller). I don't know if that'll work, the connector you linked is for hooking a SATA to an SAS Controller.
|
|
#
?
Aug 9, 2017 00:18
|
|
|
CommieGIR posted:I don't know if that'll work, the connector you linked is for hooking a SATA to an SAS Controller. Well, this is exactly my use-case: quote:The SFF-8482 to SATA Cable with LP4 Power is designed to connect one Serial Attached SCSI (SAS) hard disk drive to a SAS compatible Serial ATA (SATA) controller.  If I screwed up oh well, it's $8. Looks right to me though. edit: controller lspci's as: LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS (rev 08), and it has four SATA-type plugs IIRC. Probably something roughly similar to this. Paul MaudDib fucked around with this message at 01:55 on Aug 9, 2017 |
|
#
?
Aug 9, 2017 01:47
|
|
|
As long as it has the SAS connectors on the board, yes, that'll work just fine. But if it only has SATA, then its a different story.
|
|
#
?
Aug 9, 2017 03:18
|
|
|
Have any of you guys virtualized a NAS which used DAS storage as a backing? I have a Dell R620 and an MD1200 loaded with ~24TB of total pre-RAID storage. My goal is to deprecate my old WD 4 bay NAS. I'd like to use this NAS as SMB and NFS storage, mainly because I'd like to use some of it as an NFS datastore for the ESXi host. I guess I can approach it in one of two ways: Passthrough JBOD: - Set up PCIe RAID card as passthrough in ESXi - Create NAS VM with direct access to passthrough JBOD and virtualized 10GB connection - Create NFS share which I then mount on the ESXi host and use as a datastore For-realsies-RAID: - Buy an H800 and set up a proper RAID - Create a VMFS datastore from the entire MD1200 and give that to ESXi - Still create NAS VM for SMB shares and everything else I want to use a NAS for So the pros of JBOD are that I get whatever features of ZFS are, and that I probably don't need to resort to Dell tools to manage disk failures and replacements since everything is just passed through to the VM. The cons of the JBOD are that I'm doing some weird inception poo poo with ESXi. I'm virtualizing the infrastructure hosting a datastore which will host other VMs (not the NAS VM itself obviously) which is probably allright but means I have to pay (more) attention to startup sequences so I don't accidentally try to spin up something on the virtualized datastore before that datastore's infrastructure is booted up. Also like ten virtual points of failure. Problems with the vSwitch or NIC or anything will essentially cause any VM hosted on that Datastore to disappear. The pros of the RAID are that I don't have to worry much about infrastructure. ESXi will always have its datastore as long as the H800 isn't up in smoke, and I can still create a virtual NAS for storage of all my non-ESXi poo poo. The cons of the RAID are that I have to explicitly define how much of this storage I'm going to dedicate to ESXi and how much I'll dedicate to the NAS. With the JBOD I just have a pool of storage and if ESXi VMs use 90% of that then I have 10% left for NAS shares. If I use RAID I have to define how much of the storage will be used for NAS vs ESXi out of the gate without going into overprovisioning scenarios. My goal is basically set it and forget it so I'm kind of trying to get feelers out for what your guys recommendation would be. I probably forgot to mention this but the goal is to use this in a home environment so most of the storage is media/backups, and a lot of lab VMs. I also have a QNap TS-853U-RP enclosure collecting dust that I can press into service instead, but it doesn't take SAS disks as far as I know, and I've heard some iffy things about QNap's OS. edit: Also a Dell R710 I can use as a bare-metal NAS if I wanted to go down that route. some kinda jackal fucked around with this message at 15:01 on Aug 9, 2017 |
|
#
?
Aug 9, 2017 14:46
|
|
|
quote:I second the call for iSCSI. Can someone recommend the cheapest 10Gb ethernet adapters?
|
|
#
?
Aug 9, 2017 15:01
|
|
|
I can't speak whether it translates on that level, too, but going from Intel 10GbE SFP+ to Mellanox 40GbE QSFP doubled my Q1 random IO performance. Which is odd, because the actual values were going from 14MB/s to 34MB/s, which is a far cry from either 10 or 40 gigabit. So Mellanox seems to do some magic in the drivers (interrupt moderation, batching, whatever else). So I'd say, definitely don't go with Intel. Mellanox ConnectX2 10GbE cards can already be had for 25-30$ a piece on eBay. Recent WinOF drivers don't list them anymore as supported, but from what I keep reading, they still work with the card. However as far as cabling goes, passive SFP+ copper cables only work up to 7 meters. There's a bunch of 10 meter out of spec ones, but whether they work is more a thing of chance. Beyond that, you'll need fiber transceivers. Not sure what they cost, but last I remember, they seemed cheap enough used. --edit: I guess 10GbE on RJ45 would do. Not sure what's there used. New you'll get these Aquantia cards, which are the hot poo poo currently, for 100bux a piece. I think max. cable length is only 35m though. Combat Pretzel fucked around with this message at 18:37 on Aug 9, 2017 |
|
#
?
Aug 9, 2017 18:03
|
|
|
apropos man posted:I'm using the 6100 in my server and it's great. I'm using a Gigabyte X150M-ECC motherboard, with ECC RAM. Over in the PC Build/Parts thread, someone suggested I switch to a cheaper Kaby Lake Pentium (the G4560) which should offer similar performance. It can do ECC too but a compatible board will almost double the cost of a non-ECC board. Is ECC worth it? From what I can tell, it a matter of personal preference on the risk of losing data.
|
|
#
?
Aug 9, 2017 18:40
|
|
|
Thwomp posted:Over in the PC Build/Parts thread, someone suggested I switch to a cheaper Kaby Lake Pentium (the G4560) which should offer similar performance. Yeah get a G4560 if you can find it. There's no reason to go with the i3 unless you need the slightly faster graphics (in which case you go with the G4620 instead). Intel is reportedly choking down on G4560 production though, they're harder to find and prices are up. ECC is another layer in your defenses against data loss. You're not going to instantly lose data if you don't use it, but without it a stick of memory can quietly go bad and you don't really have any way of telling. But if the consequence of failure here is something like "you need to re-download all your animes" then yeah you could probably skip it without consequence. Just make sure all your irreplaceable data (family photos, etc) are all backed up somewhere else, which they should be anyway.
|
|
#
?
Aug 9, 2017 18:53
|
|
|
Thwomp posted:Over in the PC Build/Parts thread, someone suggested I switch to a cheaper Kaby Lake Pentium (the G4560) which should offer similar performance. Edit: I am dumb Moey fucked around with this message at 19:12 on Aug 9, 2017 |
|
#
?
Aug 9, 2017 18:54
|
|
|
Moey posted:Almost 2k less passmark score, so depending of what all you are doing with it, it may matter. I show a G4560 has a passmark of 5089 and an i3-6100 has 5481? Paul MaudDib posted:Yeah get a G4560 if you can find it. There's no reason to go with the i3 unless you need the slightly faster graphics (in which case you go with the G4620 instead). Intel is reportedly choking down on G4560 production though, they're harder to find and prices are up. Noted.
|
|
#
?
Aug 9, 2017 18:59
|
|
|
Thwomp posted:Over in the PC Build/Parts thread, someone suggested I switch to a cheaper Kaby Lake Pentium (the G4560) which should offer similar performance. ECC is not necessary. If you want to save the money just ignore it.
|
|
#
?
Aug 9, 2017 19:02
|
|
|
Thwomp posted:I show a G4560 has a passmark of 5089 and an i3-6100 has 5481? Whoa, no idea what I pulled a score of, but you are right. Ignore me.
|
|
#
?
Aug 9, 2017 19:12
|
|
|
Matt Zerella posted:ECC is not necessary. If you want to save the money just ignore it. Alternatively, consider picking up a 1-generation-old Supermicro board and a CPU to go with it, lightly used on eBay.
|
|
#
?
Aug 9, 2017 19:16
|
|
|
IOwnCalculus posted:Alternatively, consider picking up a 1-generation-old Supermicro board and a CPU to go with it, lightly used on eBay. Yeah I mean if you don't give a poo poo about size there are tons of options, ranging from the standard recommendation of an off-lease TS140 tower to repurposing an old CAD workstation. All of those are going to be a lot bigger than an Elite 110 case, and they will consume a lot of power, but they are cheap and will be very decent performance-wise. You can also just get an off-the-shelf Synology. Yeah they aren't as powerful as a dedicated rig but they're small and what they have specced out in that build isn't particularly powerful either. The main limitations of the Synology units are that the non-plus units (eg DS1817+ vs DS1817) have pretty weak CPUs, the RAM capacity is very limited (typically two sticks for 16 GB max), and you only get a single PCIe 2.0x4 expansion slot (assuming you get any slots at all). If you want to do a small-but-powerful DIY build this is what I have specced out, with a used processor (~$250) and probably half the RAM ($63 a stick from Directron, PCPP doesn't show stock right for them). If you want to cheapen it up you could go with a random LGA1151 board and the G4560 (which would in turn give you a better selection of coolers, that cooler is not ideal, just least bad), the only catch would be then you'd need a SATA controller card too, which would eat some of your savings. You could drop the replacement Noctua fans too. Regardless this is definitely quite a bit spendier than your build, but you'll have vastly greater expansion options. (Note that my build does not have video outputs, you will need to either install using a GPU and then switch to your final config, use the onboard serial output, or do a "headless install" of some kind. Also, I have not actually done this build but on paper those components are compatible.) Paul MaudDib fucked around with this message at 19:43 on Aug 9, 2017 |
|
#
?
Aug 9, 2017 19:36
|
|
|
Just an FYI but ECC RAM can result in MORE crashes in that uncorrectable writes or reads that fail after so many retries will be reported to the kernel and can result in a kernel panic because data safety matters more than systen uptime in an ECC system. My UDIMM based system has certainly rebooted more than my standard RAM systems and if I look in BIOS logs I can see a record of times an uncorrectable memory transaction occurred and those will usually correspond to times I noticed the system reboot on its own. Note that the ZFS devs at Oracle (and formerly of Sun / Oracle) are on record that ZFS does not have greater risks of data corruption from lack of ECC than any other software RAID. I personally recommend ECC because it's a nominal cost for most builds and most of the price premiums of such builds are around server grade motherboards anyway.
|
|
#
?
Aug 10, 2017 01:38
|
|
|
If you're getting that many kernel panics from ecc ram it's just a bad stick. It's not that it never happens but cosmic rays or other completely random faults are only going to tend to do enough to flip a single bit at a time which ecc will correct for. If you're getting any double errors something is really wrong. There's no need for ecc with zfs but I would say anyone rolling their own NAS should build a system with ecc regardless of the file system they intend to use.
|
|
#
?
Aug 10, 2017 03:25
|
|
|
Desuwa posted:If you're getting that many kernel panics from ecc ram it's just a bad stick. It's not that it never happens but cosmic rays or other completely random faults are only going to tend to do enough to flip a single bit at a time which ecc will correct for. If you're getting any double errors something is really wrong. Pretty much this. I've had ECC systems that have run for years without a random reboot from RAM errors. If you're getting them often enough to notice, you've got hardware failure.
|
|
#
?
Aug 10, 2017 04:23
|
|
|
If availability is what you're after, and it typically is when it comes to servers, there's a few pieces of scientific research that point to ECC being almost-required (and that it lack is a bigger problem than some people seem to think), which seems obvious when you know that ECC is one of the things referred to under the umbrella of RAS, which exist to ensure high-availability. The standard ECC memory you get for the type of servers usually brought up in this thread can only correct for one error in the entire DIMM at a time, whereas other types of ECC can protect against multiple errors on the same chip (Chipkill, Extended ECC, Advanced ECC, Lockstep Memory), as well as the loss of entire DIMMs (Redundant Array of Independent NAND and RAISE level 2). Matt Ahrens on Ars' forums posted:There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error . I personally interpret that to mean "use ECC for any system which contains data you really care about, regardless of whether it's using ZFS or not, and use ZFS if you're serious about it". BlankSystemDaemon fucked around with this message at 07:18 on Aug 10, 2017 |
|
#
?
Aug 10, 2017 07:00
|
|
|
The whole idea that you must use ECC with ZFS, and that you shouldn't use ZFS without ECC, to the point where people say you'd be better off using something else, is rooted in the myth of the "scrub of death." It's simply not the case that bad RAM can cause ZFS to overwrite good data with bad; that is, unless it's both actively malicious and intelligent, in which case you should just give up.
|
|
#
?
Aug 10, 2017 07:32
|
|
|
Speaking of the scrub of death and other myths, which it's long speculated comes from peoples lack of understanding of (and ability to read) the source code - it's the subject on one of the newer episodes of BSDNow where Matt Ahrens apparently contacted Allan Jude with an overview of the scrub implimentation: https://www.youtube.com/watch?v=rA6YmkxD6QQ&t=151s BlankSystemDaemon fucked around with this message at 08:13 on Aug 10, 2017 |
|
#
?
Aug 10, 2017 08:11
|
|
|
So, given that everyone says ECC is not necessary for ZFS, does that mean it'll checksum the data as soon as it hits the in-memory write buffers already? While it's unlikely that a cosmic ray will tumble a bit during the short window between buffering and actual write to disk, there's also bad memory cells. And what about the ARC? Is it checksummed? I can't remember, and a quick google for it always comes up with L2ARC. If the ARC _isn't_ checksummed (a block's checksum resides in the block above it in the B-tree hierarchy, you can't cache all blocks and branches to the uber one for checksumming, so I'm not sure), hot data that lingers for too long can go bad.
|
|
#
?
Aug 10, 2017 11:41
|
|
|
Matt Ahrens' answer from the show notes posted:An undetected memory error could change the in-memory checksum or data, causing ZFS to incorrectly think that the data on disk doesn’t match the checksum. In that case, ZFS would attempt to repair the data by first re-reading the same offset on disk, and then reading from any other available copies of the data (e.g. mirrors, ditto blocks, or RAIDZ reconstruction). With regards to stuff being kept in ZIL between write sequences, that's only asynchronous writes. If a write is important enough that it needs to be on disk immediately, all you have to do is make a synchronous write (either by ensuring that the application you're using issues synchronous writes, or using a dataset with the property sync=always). Add to that that ZFS is atomic, which means that it doesn't get rid of its last consistent state before a new one is completed, and you begin to understand why ZFS is so well-liked. BlankSystemDaemon fucked around with this message at 12:42 on Aug 10, 2017 |
|
#
?
Aug 10, 2017 12:35
|
|
|
|
| # ? May 12, 2024 17:08 |
|
|
Please use ECC RAM. It really does make a significant impact to data integrity and will prevent downtime. I've had machines run for 5-6 years without error due to it, whereas I've had desktops crash and burn repeatedly due to errors that ECC would have caught. That's also before you get into that boards designed to use ECC RAM are also generally designed to be fault tolerant of DIMM failures. If data integrity is not a big thing for your home NAS, fine, its up to you, but frankly its a worthwhile investment. As for in a datacenter environment, it used to be you could get away without ECC due to failover ability and you had more metal available to failover to, however, due to increasing consolidation due to virtualization, that's not as common anymore. CommieGIR fucked around with this message at 13:21 on Aug 10, 2017 |
|
#
?
Aug 10, 2017 13:09
|
|