Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
movax
Aug 30, 2008

I take it they will be 4K sectors probably?

Adbot
ADBOT LOVES YOU

FCKGW
May 21, 2006

The PC Perspective article says that it's 4k and that TLER is enabled

http://pcper.com/news/Storage/Western-Digital-releases-Red-series-SOHO-NAS-hard-drives

FCKGW fucked around with this message at 22:58 on Jul 10, 2012

Star War Sex Parrot
Oct 2, 2003

movax posted:

I take it they will be 4K sectors probably?
Yep. Just about every new consumer-grade drive should be 4K sector by now. Enterprise drives are still frequently shipping with 512 and 520-byte sectors.

Longinus00
Dec 29, 2005
Ur-Quan
If you're using these in a raid style environment as is intended for the product then so long as stripe size is some multiple of sector size it really doesn't matter what the sector size is. Assuming, of course, your controller/OS isn't retarded.

movax
Aug 30, 2008

I don't know if anyone else is on zfs-discuss, but it's been blowing up like crazy for the past 24h. Some guy talking about his benchmarks/speed improvements for dedup by trying various hashing algorithms.

Sašo Kiselkov posted:

Hi guys,

I'm contemplating implementing a new fast hash algorithm in Illumos' ZFS
implementation to supplant the currently utilized sha256. On modern
64-bit CPUs SHA-256 is actually much slower than SHA-512 and indeed much
slower than many of the SHA-3 candidates, so I went out and did some
testing (details attached) on a possible new hash algorithm that might
improve on this situation.

However, before I start out on a pointless endeavor, I wanted to probe
the field of ZFS users, especially those using dedup, on whether their
workloads would benefit from a faster hash algorithm (and hence, lower
CPU utilization). Developments of late have suggested to me three
possible candidates:

* SHA-512: simplest to implement (since the code is already in the
kernel) and provides a modest performance boost of around 60%.

* Skein-512: overall fastest of the SHA-3 finalists and much faster
than SHA-512 (around 120-150% faster than the current sha256).

* Edon-R-512: probably the fastest general purpose hash algorithm I've
ever seen (upward of 300% speedup over sha256) , but might have
potential security problems (though I don't think this is of any
relevance to ZFS, as it doesn't use the hash for any kind of security
purposes, but only for data integrity & dedup).

My testing procedure: nothing sophisticated, I took the implementation
of sha256 from the Illumos kernel and simply ran it on a dedicated
psrset (where possible with a whole CPU dedicated, even if only to a
single thread) - I tested both the generic C implementation and the
Intel assembly implementation. The Skein and Edon-R implementations are
in C optimized for 64-bit architectures from the respective authors (the
most up to date versions I could find). All code has been compiled using
GCC 3.4.3 from the repos (the same that can be used for building
Illumos). Sadly, I don't have access to Sun Studio.

Longinus00
Dec 29, 2005
Ur-Quan

movax posted:

I don't know if anyone else is on zfs-discuss, but it's been blowing up like crazy for the past 24h. Some guy talking about his benchmarks/speed improvements for dedup by trying various hashing algorithms.

All I see in the thread are people arguing about the merits of worrying about hash collisions with sha256. One side is bringing up the fact that the probability is exceedingly small, the other side is saying that people win lotteries.

b0lt
Apr 29, 2005

quote:

Since there is a finite number of bit patterns per block, have you tried to just calculate the SHA-256 or SHA-512 for every possible bit pattern to see if there is ever a collision? If you found an algorithm that produced no collisions for any possible block bit pattern, wouldn't that be the win?

quote:

If no one has computed the hashes for every single 4K and 8K block, then fine. But, if that was done, and we had that data, we'd know for sure which algorithm was going to work the best for the number of bits we are considering.

Speculating based on the theory of the algorithms for "random" number of bits is just silly. Where's the real data that tells us what we need to know?

:laugh:

Goon Matchmaker
Oct 23, 2003

I play too much EVE-Online
I'm no math expert but even I can understand the likeihood of a hash collision is so infinitesimally small that I'm more likely to witness the sudden supernova of every star in the observable universe. That Gregg guy must be retarded. I am, however, extremely amused by that thread.

evil_bunnY
Apr 2, 2003

Yeah I really wish the guys who don't understand no one is going to compute 2^(8x4096) hashes would just be quiet.
Also you can tell ZFS to look at the actual blocks when it finds a dedup hit (and you should).

movax
Aug 30, 2008

Longinus00 posted:

All I see in the thread are people arguing about the merits of worrying about hash collisions with sha256. One side is bringing up the fact that the probability is exceedingly small, the other side is saying that people win lotteries.

It descended into madness real quick, but I thought it was interesting from his original e-mail:

Sašo Kiselkov posted:

* SHA-512: simplest to implement (since the code is already in the
kernel) and provides a modest performance boost of around 60%.

* Skein-512: overall fastest of the SHA-3 finalists and much faster
than SHA-512 (around 120-150% faster than the current sha256).

* Edon-R-512: probably the fastest general purpose hash algorithm I've
ever seen (upward of 300% speedup over sha256) , but might have
potential security problems (though I don't think this is of any
relevance to ZFS, as it doesn't use the hash for any kind of security
purposes, but only for data integrity & dedup).

My testing procedure: nothing sophisticated, I took the implementation
of sha256 from the Illumos kernel and simply ran it on a dedicated
psrset (where possible with a whole CPU dedicated, even if only to a
single thread) - I tested both the generic C implementation and the
Intel assembly implementation. The Skein and Edon-R implementations are
in C optimized for 64-bit architectures from the respective authors (the
most up to date versions I could find). All code has been compiled using
GCC 3.4.3 from the repos (the same that can be used for building
Illumos). Sadly, I don't have access to Sun Studio.

Not super detailed about the test conditions and such, but even then, those are pretty sweet numbers.

IOwnCalculus
Apr 2, 2003





evil_bunnY posted:

Yeah I really wish the guys who don't understand no one is going to compute 2^(8x4096) hashes would just be quiet.
Also you can tell ZFS to look at the actual blocks when it finds a dedup hit (and you should).

Seriously, I don't see why you couldn't use a much faster method, even if it does result in the potential for hash collisions, as long as you tell it to verify that the two blocks are indeed identical. I can't imagine that last step would take more time / resources than a 60-300% increase in the hash speed itself would save.

Longinus00
Dec 29, 2005
Ur-Quan

movax posted:

It descended into madness real quick, but I thought it was interesting from his original e-mail:


Not super detailed about the test conditions and such, but even then, those are pretty sweet numbers.

If you want to do a silly test just time the difference between running sha256sum and sha512sum on a large file. Don't forget to cat the file to /dev/null first to get it into disk cache.

code:
#!/bin/sh

cat $1 > /dev/null
time sha256sum $1 > /dev/null
time sha512sum $1 > /dev/null

Bensa
Aug 21, 2007

Loyal 'til the end.
Will connecting a boot disk pulled from an AHCI mode system to a IDE mode one as a secondary cause any issues? If msahci doesn't load when you connect one it could explain why I wasn't able to access the disk and diagnostics kept on giving errors.
Just ran into an issue where a WD Black kept on throwing errors while in IDE mode but runs perfectly in AHCI. The errors while in IDE mode persisted through zeroing the drive.

Factory Factory
Mar 19, 2010

This is what
Arcane Velocity was like.

Bensa posted:

Will connecting a boot disk pulled from an AHCI mode system to a IDE mode one as a secondary cause any issues? If msahci doesn't load when you connect one it could explain why I wasn't able to access the disk and diagnostics kept on giving errors.
Just ran into an issue where a WD Black kept on throwing errors while in IDE mode but runs perfectly in AHCI. The errors while in IDE mode persisted through zeroing the drive.

As secondary? No, as long as the controller itself has drivers installed, in the OS doesn't give a crap about secondary disks because nothing boot-time critical happens on them.

Were you hot-plugging the drive? IDE can't hot plug but AHCI can, and that might make the difference. Otherwise I'd be looking at the drive as not quite right.

Bensa
Aug 21, 2007

Loyal 'til the end.
I think I might have hot-plugged it once or twice (not exactly sure, been working on multiple disks) while still in IDE, but only after the issues started popping up and I tried to troubleshoot it. I only hot-plugged the data connector and then I used device manager to get windows to recognize it. To me it would seem very odd that this would cause damage only noticeable in IDE mode.
The issues in IDE mode presented in the Windows and DOS versions of WD DLG. The DOS version tests (and zeroing) refuse to even run while in IDE mode, the Windows version tests either can't finish or report bad sectors which it manages to correct. The remapped sector count is still zero. After I zeroed the drive while in AHCI mode there have been no issues while staying in AHCI.

The course of events:
Install WD Black
Clone OS disk to WD with Acronis
Try to boot from WD, NTFS errors
Boot on old disk, Windows acts really off while WD is active
Run WD DLG in DOS, still runs but with errors
Start diagnostics, zeroing etc, keep on finding issues (hot-plug once or twice?)
WD DLG in DOS no longer runs
Switch to AHCI and zero, no more issues in both Windows and DOS (while in AHCI)

necrobobsledder
Mar 21, 2005
Lay down your soul to the gods rock 'n roll
Nap Ghost
For the purposes of data integrity validation in ZFS's context on anything but data gathered from radiation or something for longer than the known age of the universe2048, it's just fine and dandy for dedupe even as an exabyte capable filesystem. SHA's big draw is that its uniformity is well understood, it's extensible (SHA2-nm isn't necessarily possible nor secure with every algorithm) it's fairly easy to implement, and balances with implementation speed. The few problems you could have are issues with key extension attacks that could conflict with the data elsewhere in the filesystem (somewhat akin to trying to do a file pointer substitution attack).

The bigger concern to me would be compatibility of the on-disk format. Is ZFS's on-disk format cool enough that you can self-describe the hashing algorithm as a fat binary library or something? That'd be kind of interesting I guess as a pet project for the persistent data crypto spergs.

I seriously don't get what's with people posting criticisms about hash algorithms that have pretty obviously never even looked at the math beyond some casual CS course or brief Wikipedia article. Furthermore, if you have such crazy and paranoid needs, you would be using dedupe and verification at multiple layers rather than relying solely upon your file system's dedupe.

evil_bunnY posted:

Also you can tell ZFS to look at the actual blocks when it finds a dedup hit (and you should).
The verify option could make dedupe so slow I'd argue you're almost better off not bothering with dedupe if you're that paranoid.

UndyingShadow
May 15, 2006
You're looking ESPECIALLY shadowy this evening, Sir
All this talk about Red Drives and TLER has me concerned. I built my NAS box using an NL40 and 6 5400 rpm green drives (5 Samsung and 1 WD Green). I'm using the latest build of FreeNAS 8 with zfs

Do I need to worry about my drives dropping out of the array? I haven't had any problems so far, but I'm concerned.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

UndyingShadow posted:

Do I need to worry about my drives dropping out of the array? I haven't had any problems so far, but I'm concerned.
Not really. If you were setting up a new system today, it might be worth a few bucks per drive to go with Red, but it's certainly not worth replacing current drives.

thideras
Oct 27, 2010

Fuck you, I'm a tree.
Fun Shoe

UndyingShadow posted:

All this talk about Red Drives and TLER has me concerned. I built my NAS box using an NL40 and 6 5400 rpm green drives (5 Samsung and 1 WD Green). I'm using the latest build of FreeNAS 8 with zfs

Do I need to worry about my drives dropping out of the array? I haven't had any problems so far, but I'm concerned.
As DrDork said, you will likely be fine. Consumer level TLER usually only has problems when the disks will be dropped under 90 seconds. Most software RAID will just wait for the disk to return. A hardware RAID controller will fail the drive around 9 seconds.

b0lt
Apr 29, 2005

necrobobsledder posted:

The bigger concern to me would be compatibility of the on-disk format. Is ZFS's on-disk format cool enough that you can self-describe the hashing algorithm as a fat binary library or something? That'd be kind of interesting I guess as a pet project for the persistent data crypto spergs.

This is the main problem, I think. The reason dedup was relatively easy to drop into ZFS is that the checksum for blocks in the original ZFS on disk format was 256 bits. You could do something like calculate SHA-512 and then truncate to fit to get the performance advantage of using SHA-512 on 64bit (see SHA-512/256), but that might be seen as too clunky.

DrDork
Dec 29, 2003
commanding officer of the Army of Dorkness

thideras posted:

Consumer level TLER usually only has problems when the disks will be dropped under 90 seconds. Most software RAID will just wait for the disk to return. A hardware RAID controller will fail the drive around 9 seconds.
Yeah, pretty much. Since you're using a N40L you don't have hardware RAID, so all FreeNAS/ZFS is going to do if you get a drive that sticks on recovering an error for a bit is a nasty email (if you've set that up) while it's degraded, and then once the drive pops back ZFS will pick it up and it'll be all good again. The real concern with TLER and hardware RAID is on systems where a degraded array automatically kicks off a recovery process, which doesn't really apply to you.

Delta-Wye
Sep 29, 2005
Fellow packrats, if you were buying new drives to upgrade your NAS, would you be willing to go with one of these green drives?

http://www.amazon.com/dp/B004VFJ9MK/

I understand the line between "don't bother replacing, but don't buy new". The price is certainly right, but they are WD green drives. Right now my ZFS-based system has 4 320GB drives and it's time to expand. I think at least a couple are already WD green drives and I haven't seen much in the way of problems with them. At $100 a piece, I can stand to get 4 right now and triple my storage space.

If there is something more suitable I'd be willing to consider it. Those new Red drives look fancy, but I'm sure they'll be more $$$ and I'm not sure if I'll really need the features.

FCKGW
May 21, 2006

Delta-Wye posted:

Fellow packrats, if you were buying new drives to upgrade your NAS, would you be willing to go with one of these green drives?

http://www.amazon.com/dp/B004VFJ9MK/

I understand the line between "don't bother replacing, but don't buy new". The price is certainly right, but they are WD green drives. Right now my ZFS-based system has 4 320GB drives and it's time to expand. I think at least a couple are already WD green drives and I haven't seen much in the way of problems with them. At $100 a piece, I can stand to get 4 right now and triple my storage space.

If there is something more suitable I'd be willing to consider it. Those new Red drives look fancy, but I'm sure they'll be more $$$ and I'm not sure if I'll really need the features.

Newegg has the Red drives in stock now, looks like about $10 premium over their 2TB Green ($30 over your Amazon link).
http://www.newegg.com/Product/Product.aspx?Item=N82E16822236343

PcPer has a full review now too http://www.pcper.com/reviews/Storage/Western-Digital-Red-3TB-SATA-SOHO-NAS-Drive-Full-Review/Conclusion-Pricing-and-Thoug

FCKGW fucked around with this message at 19:35 on Jul 13, 2012

Krakkles
May 5, 2003

Picked up 4 of these. With coupon code EMCNCJN23, they're $99 a piece, I figured that was a good price.

Now just waiting for a deal on an N40L and WHS2011.

Delta-Wye
Sep 29, 2005

Krakkles posted:

Picked up 4 of these. With coupon code EMCNCJN23, they're $99 a piece, I figured that was a good price.

Now just waiting for a deal on an N40L and WHS2011.

Those are the same drives I'm asking about, at the same price without prime shipping :ssh:

The price does seem pretty damned good though, as long as you don't look too far back:
http://camelcamelcamel.com/Western-Digital-Caviar-Desktop-WD20EARX/product/B004VFJ9MK

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Krakkles posted:

Picked up 4 of these. With coupon code EMCNCJN23, they're $99 a piece, I figured that was a good price.

Now just waiting for a deal on an N40L and WHS2011.

The 3TB Seagate ST3000DM001 is also $50 per TB right now. I'm still holding out for a return to the $30/TB glory days.

Krakkles
May 5, 2003

Delta-Wye posted:

Those are the same drives I'm asking about, at the same price without prime shipping :ssh:

The price does seem pretty damned good though, as long as you don't look too far back:
http://camelcamelcamel.com/Western-Digital-Caviar-Desktop-WD20EARX/product/B004VFJ9MK

drat. Ok, cancelled with Newegg, ordered from Amazon. Saved ~$35 on tax. Thank you!

IOwnCalculus
Apr 2, 2003





fletcher posted:

The 3TB Seagate ST3000DM001 is also $50 per TB right now. I'm still holding out for a return to the $30/TB glory days.

Ooof. Getting tempting since that meets my minimum at-least-double-my-old-drives swap. Went from 500GBs to 1.5TBs, and after I recreated it as a RAIDZ2 instead of mdraid RAID5, I'm a bit tighter on space...

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

fletcher posted:

The 3TB Seagate ST3000DM001 is also $50 per TB right now. I'm still holding out for a return to the $30/TB glory days.

Hmm. Any known problems with these?

I need to start working on a plan to migrate my 16TB mdadm+LVM+ext4 system to something xfs based.

Alfajor
Jun 10, 2005

The delicious snack cake.
Not sure how, but found a D-Link DNS-320 for like $24. I hope it was a pricing error, and that it still goes through anyway: http://www.amazon.com/gp/product/B004SUO450/ref=ox_ya_os_product (says there's ONE left)

Civil
Apr 21, 2003

Do you see this? This means "Have a nice day".

Thermopyle posted:

Hmm. Any known problems with these?

I need to start working on a plan to migrate my 16TB mdadm+LVM+ext4 system to something xfs based.

1 yr warranty is kind of a disappointment. Other than that, hard drives are largely similar, especially if you're just using them as server fodder.

Horn
Jun 18, 2004

Penetration is the key to success
College Slice

Thermopyle posted:

Hmm. Any known problems with these?

I need to start working on a plan to migrate my 16TB mdadm+LVM+ext4 system to something xfs based.

Any reason why you've moving away from ext4? I'm getting ready to redo my stuff as well and I've been using ext3/ext4 for quite a while - at least since Rieser killed his wife.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Horn posted:

Any reason why you've moving away from ext4? I'm getting ready to redo my stuff as well and I've been using ext3/ext4 for quite a while - at least since Rieser killed his wife.

I talked about it in this post and there's some discussion in the surrounding posts.

Stein Rockon
Feb 5, 2005

SATAN SANTA TRADE YOUR SOUL FOR MY ORANGES
I'm looking for a storage solution. I'm a photographer, so I need to find a second place to store around 1TB of photos (raw-files). I've also got a load of (replaceable) videos that I would like to stream/transfer, both in house, and to my brother who is at college 970 miles away and can't acquire things the normal way while on their network.

I was originally set on buying a Netgear Readynas Ultra 4 (not the plus version) and chucking 2-3 2TB drives in it. It would cost me about $330 for the NAS, plus $140 per drive. Alternatively I can get the new 2TB WD Red drives for $170 a pop.

Then I started reading this thread (I began on page 100), and I've learned that the N40L is the preferred solution in these parts.

I've checked, and the cheapest price for the N40L is $334, plus $77 for the 2x4gb ram sticks that has been posted earlier in this thread.

So it's either $750 for the Readynas and 3x2TB harddrives, or $831 for N40L, RAM and harddrives. That's not too big a difference, and I think the N40L is a better choice.

I want:
- "backup" of my raw photo files (will also backup to Crashplan)
- streaming of video (computers/PS3s/android devices), to LAN and over the internet
- access to files, also to LAN and over the internet.
- torrent capability, preferably with remote admin/adding of torrents
- ability to upload to Crashplan's online service

The Readynas Ultra, as far as I can tell, can do this with some plugins.
Can I do the same with Freenas on an N40L? How hard would it be to set up? (I can follow instructions fine, I'm just not proficient with linux)

Also, which is preferable?
- Western Digital Caviar Green WD20EARX 64MB 2TB
- Seagate Barracuda Green ST2000DL003 64MB 2TB
- Seagate Barracuda ST2000DM001 64MB 2TB


I'm assuming the prices will look very high, so I can tell you right away that there's no point in waiting for the price to drop or offers to come along. It's not likely to become any cheaper any time soon. :norway:

Importing from abroad adds a 25% tax on anything over $33 (including the price of postage), which cuts away any savings on taking that route. Importing would also mean that I lose the protection of Norwegian consumer laws, such as the right to get replacements for faulty products within 5 years (possibly only 3 years for the harddrives).

edit: better wording

Stein Rockon fucked around with this message at 00:44 on Jul 15, 2012

el_caballo
Feb 26, 2001
Does anyone know if a DS212J has enough power to run Sabnzbd, SickBeard, CouchPotato and CrashPlan? Does loading up all of these plugins kill the processor or can it handle it?

I'm trying to decide if the 212+ is necessary.

frumpsnake
Jan 30, 2001

The sad part is, he wasn't always evil.

Thermopyle posted:

Hmm. Any known problems with these?
They do/did have overly-aggressive head parking but that seems to have been fixed in one of the more recent firmware updates.

I have 5 of them in my N40L without issues.

BlankSystemDaemon
Mar 13, 2009



Stein Rockon posted:

:words: regarding N40L with FreeNAS rather than a ReadyNAS.
You also need to add in another NIC if you don't want to run into a driver problem with the bge driver on FreeBSD for the particular on-board NIC that's used in the N40L, any NIC on this page should work as long as it's got low-profile.
That being said, the crashplan client isn't going to be easy to get running on FreeNAS - it's not officially compatible, and although people have done it on FreeBSD, that's not quite the same on FreeNAS, even with the PBI plugin system in the 8.2 release.
As for disks, I'd recommend 4x Seagate Barracuda Green ST2000DL003 64MB 2TB out of the ones you have listed.
You also need a USB flash drive to install FreeNAS on, of course.

In short: Just go with the ReadyNAS solution, unless you insist on DIY solutions. The primary advantage of DIY solutions like N40L is that they're more easily expanded later on compared to a ReadyNAS solution.

BlankSystemDaemon fucked around with this message at 09:22 on Jul 15, 2012

bobfather
Sep 20, 2001

I will analyze your nervous system for beer money

el_caballo posted:

Does anyone know if a DS212J has enough power to run Sabnzbd, SickBeard, CouchPotato and CrashPlan? Does loading up all of these plugins kill the processor or can it handle it?

I'm trying to decide if the 212+ is necessary.

I'm not sure about CrashPlan, but it will run the other listed apps without problem.

As a matter of fact though, I've set mine up to run SickBeard as a cronjob since it has an annoying tendency to wake up NAS devices overly often to do its business.

Doh004
Apr 22, 2007

Mmmmm Donuts...
Any suggestions for trying to recover data from an external harddrive that isn't recognized by Windows anymore (for free)? We have a WD Passport, and whenever we connect it windows sees it and installs whatever's necessary; however, it never shows up as a drive. When going to disk management in the computer manager, the drive is listed but when trying to do anything with it we get an error saying it failed to initialize.

WD support suggests trying to get the files off it with some app.

Adbot
ADBOT LOVES YOU

Inspector_666
Oct 7, 2003

benny with the good hair

Doh004 posted:

Any suggestions for trying to recover data from an external harddrive that isn't recognized by Windows anymore (for free)? We have a WD Passport, and whenever we connect it windows sees it and installs whatever's necessary; however, it never shows up as a drive. When going to disk management in the computer manager, the drive is listed but when trying to do anything with it we get an error saying it failed to initialize.

WD support suggests trying to get the files off it with some app.

Try a Linux Live CD. If the computer sees the drive but wants to initialize it, there may be some poo poo hosed up with the formatting that just means Windows can't read it.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply