Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
KS
Jun 10, 2003
Outrageous Lumpwad

tehschulman posted:

BTW, Amazon and Newegg are selling the 3TB Seagate Barracuda for $120 down from $150. Ordered two over the weekend to fill the last 2 of my 4 bay NAS.

frumpsnake posted:

The ST3000DM001s have worked well for me, just make sure you apply the CC4H firmware update or they'll overenthusiatically park their heads.

I'm having significant problems with these drives -- they have made NAS4free, FreeBSD Stable/9, and Centos 6.3 lock up/go unresponsive. I've done the firmware update, no help.

Finally seems to be stable with FreeNAS 8.3, but figured I'd chime in. It's sucked up a few weeks of my work-life.

System is a SuperMicro SC847, 4 LSI9211s, and 32 of the ST3000DM001s with a ZeusRAM as ZIL.

Adbot
ADBOT LOVES YOU

KS
Jun 10, 2003
Outrageous Lumpwad
I'm getting really frustrated by a ZFS project and there seem to be a lot more ZFS users here than in the Enterprise Storage thread.

I started with NAS4FREE 9.1.0.1 (BSD Stable/9 based) and had stability problems where the entire system would go unresponsive. Based on some correspondence with other Stable/9 users seeing the same problem, I moved to the Stable/8-based just-released FREENAS 8.3. Stability problems are fixed -- system has been up 8 days, and both local disk performance and NFS performance to a linux client is outstanding.

However, write performance on both NFS to windows and iscsi to windows is horrible under FreeNAS. Here's what I'm looking at over 10gbit iscsi:



Obviously one of these is going to be fine as a target for disk backups, and the other one is not.

Any suggestions? Do I need to go with an installable freebsd 8 so I can update istgt? Are there tweaks that NAS4Free implements that FreeNAS does not? Is my ZIL on my imported pool maybe not working? Should I just buy Nexenta so this is someone else's problem?

The system is a SuperMicro SC847, 2x E5620, 96 GB RAM, 10gbit Intel x520-DA2 NIC, 32x ST3000DM001 3TB, and a ZeusRAM for ZIL. The pool config:

code:
        NAME        STATE     READ WRITE CKSUM
        Pool        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da32    ONLINE       0     0     0
            da22    ONLINE       0     0     0
            da29    ONLINE       0     0     0
            da30    ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da17    ONLINE       0     0     0
          raidz2-1  ONLINE       0     0     0
            da16    ONLINE       0     0     0
            da15    ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0
            da12    ONLINE       0     0     0
            da13    ONLINE       0     0     0
            da7     ONLINE       0     0     0
          raidz2-2  ONLINE       0     0     0
            da14    ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da18    ONLINE       0     0     0
            da19    ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da20    ONLINE       0     0     0
            da5     ONLINE       0     0     0
          raidz2-3  ONLINE       0     0     0
            da31    ONLINE       0     0     0
            da23    ONLINE       0     0     0
            da26    ONLINE       0     0     0
            da27    ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da24    ONLINE       0     0     0
            da21    ONLINE       0     0     0
            da28    ONLINE       0     0     0
        logs
          da25      ONLINE       0     0     0

KS fucked around with this message at 00:55 on Nov 9, 2012

KS
Jun 10, 2003
Outrageous Lumpwad
$272 shipped, thanks for posting! Been waiting for another N40L deal for a while.

KS
Jun 10, 2003
Outrageous Lumpwad
Well, after buying an N40L, I think I'm going to look for something more powerful. My ECC RAM is registered and won't work, and I think I can get something more powerful for the same money given that I'd need to re-buy the RAM.

My goal is a 32GB machine with enough power to act as an ESX server, and VT-D support to pass the storage through to a ZFS VM.

I'm looking for a case and PSU recommendation that's similar size to the N40L, quiet, and can take 4 3.5" drives and an SSD. I'm going to put a Supermicro x8SIL-F in it since the newer X9s also only take unbuffered RAM, unless someone has a better suggestion.

I'm going to try to sell the N40L on SA Mart, I guess -- when did Newegg start charging a 15% restocking fee? I don't think I'm going to shop anywhere but Amazon from now on.

KS
Jun 10, 2003
Outrageous Lumpwad
The CD version of both FreeNAS and Nas4Free include an option once booted that will do the install to another device, including a key drive. It seems to be the easiest option.

KS
Jun 10, 2003
Outrageous Lumpwad
Just a heads up, N40L is $259.99 and free shipping at Newegg right now. It rarely goes this low, and it hasn't since August.

You can also get a combo deal to add a single 3TB ST3000DM001 hard drive for $80, which is pretty good.

KS
Jun 10, 2003
Outrageous Lumpwad
Just a thought, but I can't imagine you're ever going to make back the cost of that SSD in electricity savings if you're just talking about a few watts.

KS
Jun 10, 2003
Outrageous Lumpwad
Does anyone know if I can replace ST3000DM001 drives with WD30EFRX drives in a ZFS array using whole disk devices? I'm concerned there might be slight capacity differences that would prevent this from working. Is the relevant specification "user sectors per drive?" They look identical at 5,860,533,168.

I just had a double drive failure in a RAID-Z2 of 6 ST3000DM001 drives. I'm leery of just RMAing due to the somewhat bad reputation this model is getting, but if I buy replacements I want them to work.

KS
Jun 10, 2003
Outrageous Lumpwad

insularis posted:

Which should I use at this point? NAS4free, FreeNAS, ZFSonLinux and set it up manually? Recommended motherboard?

Advice?

I know this is the home thread, but you're talking about a work system, so my advice is Illumos. I had stability issues with Nas4Free and performance issues with FreeNAS before biting the bullet and going for something Solaris-based. It paid off -- the system is both fast and rock solid. I find the documentation is way better as well, because about 95% of it agrees with the Oracle docs still. Some things, like disabling cache flush on a per-device basis, don't even seem to be implemented on FreeBSD.

This is on a 32x 3TB array with a ZeusRAM and all the goodies. Nas4free runs perfectly fine at home on my little 6 disk setup. Your mileage may vary.

KS
Jun 10, 2003
Outrageous Lumpwad
My crashplan situation's getting worse -- I have it on a VM that has my big file system NFS mounted, and it's worked great for 6 years, but the VM's started hard locking after I upgraded it to Centos 6.10.

My current solution is to just reboot it whenever crashplan notifies me it's disconnected -- probably once a week or so. I'd like to rebuild it, but my understanding is the newer client versions may/may not run headless and definitely don't let you turn off dedupe, which is the only way backup speeds are acceptable. On an 8TB dataset, speeds were down to <10KB/sec with dedupe on but saturated my internet connection once turned off.

Guess I'm just wondering if there's anything else out there that would handle 8TB for $10/mo or so.

KS
Jun 10, 2003
Outrageous Lumpwad
Have it signal the shutdown above the level that it beeps, problem solved. That's usually configurable.

KS
Jun 10, 2003
Outrageous Lumpwad
I had the misfortune of buying 48 of the ST3000DM001s featured in Backblaze's first data set and it wasn't anything unique to Backblaze. That model was defective from the factory. That doesn't mean anything about other Seagate drives, but their response to building a defective product was to ignore it, and that does mean something.

KS
Jun 10, 2003
Outrageous Lumpwad

movax posted:

Interesting, I have been very behind on keeping up with this thread and didn’t realize that was a thing. I have an ESXi box that I’ve been meaning to put FreeNAS and other OSes on (probably a Fedora VM to run Usenet / other SW / things that want Linux, not BSD) and was going to use HW pass through of my HBAs. No longer a good idea, wasn’t ever a good idea?

Chiming in that I've been passing a 9211-8i through to a FreeNAS VM for 6 years without a single issue. OK, one -- long ago I couldn't give the VM more than a single CPU without BSD kernel panics, but that was fixed years ago in a FreeNAS upgrade. I have a half dozen linux VMs mapping the ZFS pool via NFS and doing PLEX/backups/Usenet/etc. It's been outstanding.



Guts were salvaged out of a DL380 and are in a silent case with Noctua coolers. I need to upgrade as my CPU isn't supported by new versions of ESXi any more, but :effort: on researching a cost effective replacement.

KS
Jun 10, 2003
Outrageous Lumpwad

KKKLIP ART posted:

No it isn't being a dick and is a fair question. I know that a raidZ2 gives me 2 drives that could drop. I've never had a drive fail, so it was more of a "if I pull a drive, can I still access the data and work with it or will it make me swap in a drive before I can access the data again." From what it sounds like, I theoretically could pull the drive, have access to the data, toss the "spare" into a drive caddy, format it and copy the stuff, replace the drives in my NAS, and paste the stuff onto the new drives/pool in my NAS.

Actually almost asked this when you first posted, but it looked like you had just one drive fail -- you know you can just replace it and resilver the array to restore redundancy? If you're looking to upgrade for more silence or whatever, that's great -- but it looks like you have about an $80 problem.


I'm still on Crashplan and it's getting pretty dire. Speeds with 9TB were poo poo -- like capped <100kbps -- unless you manually turn off deduplication. The newest clients no longer let you manually turn off dedupe.

KS
Jun 10, 2003
Outrageous Lumpwad


Happy 7th (!) birthday to my NAS which started life as a scrapped DL380 G6. It runs ESXi and a half dozen VMs for plex/sonarr/transmission, including a xigmanas VM with hw passthrough for ZFS. It's silent, lives in a closet, and has IPMI, which is great. 7 years of trouble free operation with the exception that the first drives I put in it were ST3000DM001s. 5 died in the first year. The 6th is still running.

Antec P280 with Seasonic X650
Supermicro X8DTL-iF-O
Dual Xeon E5540 with Noctua NH-U12DX
48GB ECC
M1015 with 6x 3TB drives
some other LSI 9208 board with 2x 512GB SSDs, RAID-1, for VM storage.

My storage growth rate has increased recently. The array's nearly full and wondering what to do next. I could just swap the drives to 12TB in the same pool, but it'd be cool to move to an 8-drive array. The processor's also old enough that it's off VMware's HCL and can't run ESXi 6.7, and it's not very power efficient to be running dual CPUs by modern standards.

Feels like ESXi is getting a bit old fashioned and I'd welcome the chance to do less CJing of VMs in favor of containers, but ZFS has saved me a few times and I don't want to give that up. Any recommendations for the next 7 years?

KS
Jun 10, 2003
Outrageous Lumpwad

I had no idea, thanks.

I've been trying to figure out what to target if I want a new box and not sure I gain much --- is it correct that the current xeon E3 is 3 years old? Pondering just buying a pair of xeon E5640s on Ebay for $10 as they're apparently still on the ESXi 6.7 HCL and work in my current motherboard.

KS
Jun 10, 2003
Outrageous Lumpwad
In the last few minutes I finished moving from 6x 3TB reds to 6x shucked white label 12TBs and the pile of trash left over is disgusting from an environmental perspective. $900 less expensive vs. bare drives.

They don't appear to be SMR and they have TLER. I did 3x resilvers at about 20 hours each, and they performed as good or better as the WD30EFRXs they replaced.

KS
Jun 10, 2003
Outrageous Lumpwad

Toshimo posted:

Ok, I've got my money lined up. Where, other than Slickdeals, should I be monitoring for 12TB drives to shuck?

Camelcamelcamel alert for Amazon and checking Best Buy pretty much covers it.

KS
Jun 10, 2003
Outrageous Lumpwad
Personal bias since I did this, but I think the way to go is the RAM and CPUs from an older server, a matching Supermicro ATX board, and silence-oriented tower cases and noctua fans. Rack mount stuff is loud.

Right now 2600 series CPUs and X10 motherboards with all sorts of badass features and IPMI can be had for peanuts. 1U servers sell at a discount compared to 2u, and damaged chassis more so. Looks like you can get a motherboard, two 12-core 2667s, and 128GB ECC RAM for $550ish. That's nuts.

My earlier post started life as a DL380 G6. I recently grabbed 2 X5660 processors and 96GB RAM to give it another few years. It cost $40 after selling my old 48GB RAM set.

KS fucked around with this message at 21:48 on Mar 12, 2021

KS
Jun 10, 2003
Outrageous Lumpwad
In Debian, just use LVM.

Adbot
ADBOT LOVES YOU

KS
Jun 10, 2003
Outrageous Lumpwad
The UPS logs will have detail on why it's switching.

There are also usually knobs to tweak in the UPS settings -- like if it's undervoltage you can widen the tolerance.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply