|
Eh, I use a SSD mirror for my VMs and use rust for long archival. Seems like a good middle ground and not too expensive.
|
#
?
Feb 7, 2017 18:15
|
|
|
|
| # ? May 15, 2024 17:37 |
|
|
D. Ebdrup posted:No, you enabled dedup because you thought it was a magic pill that'd fix your troubles. It doesn't, it's a specific thing made for a specific use-case. I enabled it because I was curious to see how well it worked given my mixed usage and tendency to have random extra copies of crap in random folders. I didn't realize that once it's enabled it's very difficult to get rid of, and exactly how godawful the performance penalty would be once it had to start paging to disk. Looking back on it, it was a really good lesson on not turning poo poo on just to see what it does, because OpenSolaris and ZFS don't have a nice easy undo button for a lot of things. Given my wonky setup, I wonder how ZFS prefetch actually works. How would ZFS know what data to prefetch when the zpool is just acting as a block level device I use as an iSCSI lun? Edit: And that LUN is bitlockered. Methylethylaldehyde fucked around with this message at 18:42 on Feb 7, 2017 |
|
#
?
Feb 7, 2017 18:38
|
|
|
You have a pretty normal setup to be honest, I was running the same over Infiniband and it works really well. I don't know if prefetch would work reliably because it can't "see" inside the zvol. Maybe it will do some heuristics: "Eh, you already read 300MB sequentially you probably want some more" or something.
|
|
#
?
Feb 7, 2017 19:18
|
|
|
DrDork posted:Unless you have a fairly small NAS (at which point maybe you could get away with a simple mirroring arrangement on another computer you've got), or you really don't care whatsoever about prices, you're going to have to wait a good bit longer than two generations: No, I know how expensive they are and I know even in two years it will remain absurdly expensive, I wasn't hoping for a miracle. I was hoping another two generations would bring 4tb evo drives under $1000, though I never looked too deeply at trends.
|
|
#
?
Feb 7, 2017 20:19
|
|
|
SSD "generations" are only about a year long, though, so two more generations is about 2-ish years. In that a 4TB 850 Evo is about $1400 today, it's not irrational to believe that in two years it (or whatever 2019's 4TB SSD model is) might slip under the $1k mark.
|
|
#
?
Feb 7, 2017 21:00
|
|
|
DrDork posted:SSD "generations" are only about a year long, though, so two more generations is about 2-ish years. In that a 4TB 850 Evo is about $1400 today, it's not irrational to believe that in two years it (or whatever 2019's 4TB SSD model is) might slip under the $1k mark. My hope is that in a year or so, I can justify replacing all the laptop drives I have sitting in a 24 drive 2u shelf with $49 500gb SSDs. At that point the bottleneck would be the SAS fabric feeding them into the SAN, which obviously means I need up upgrade my Intel X520s to some 20 or 40gb infiniband setup, and spring for a 12gb HBA and backplane.
|
|
#
?
Feb 7, 2017 21:26
|
|
|
DrDork posted:SSD "generations" are only about a year long, though, so two more generations is about 2-ish years. In that a 4TB 850 Evo is about $1400 today, it's not irrational to believe that in two years it (or whatever 2019's 4TB SSD model is) might slip under the $1k mark. Yeah that was my hope. Late 2018 would put us two generations past what is currently available assuming Samsung kept to releasing at the same time each year.
|
|
#
?
Feb 7, 2017 22:03
|
|
|
Methylethylaldehyde posted:My hope is that in a year or so, I can justify replacing all the laptop drives I have sitting in a 24 drive 2u shelf with $49 500gb SSDs. At that point the bottleneck would be the SAS fabric feeding them into the SAN, which obviously means I need up upgrade my Intel X520s to some 20 or 40gb infiniband setup, and spring for a 12gb HBA and backplane. 500GB drives go for around $150 today, so hoping for them to crater to $50 in a year is probably asking a bit too much--maybe they'll be $99 this time, though. 24 * 500MBps = 12GBps = 96Gbps for the full rack, so even if you were considering a 40Gb inifiniband, you could saturate that with (in ideal conditions) as little as 10 500GB 850 Evos (or other 500MBps SSDs).
|
|
#
?
Feb 7, 2017 22:19
|
|
|
I'm currently doing my backups on an additional 1TB hard drive on my PC (not very smart i guess); I also do backups infrequently because I'm doing it manually (stupid). I'm thinking about buying a Synology DS115J to use with my existing hard drive. My original data would be stored on my PCs SSD and backed up to the 1TB hard drive which would be used by the NAS. I have two questions: -Is there anything I could do in terms of theft protection that doesn't involve me buying vaults/bear traps? -Am I missing something? I understand that I could be spending 1000$< on this, but I think a consumer grade NAS with a single drive is appropriate for me unless there's something I'm not aware of.
|
|
#
?
Feb 8, 2017 11:12
|
|
|
Otaku Alpha Male posted:I have two questions: Data is only backed up if it's backed up in two places. Doesn't matter the medium, it needs to be somewhere else. Back your critical poo poo up to Amazon Glacier. If you only have 1 TB of it then you don't have any problems, and the cost is very minimal per month. Be aware that if you need to restore it all at once the cost goes up a whole lot. Thus, Amazon Glacier may not count as a 2nd place, it's a 3rd place of last resort. quote:-Am I missing something? 1 TB is still within the amount that you can reasonably back up by yourself. Burn 25 GB BD-R discs, with 20 GB of data and a 5 GB parity file from the last disk (via DVDisaster). That way if you get multiple corrupted disks in a row you can use disk N to recover disk N-1, and all the way on back to the start of the corruption. In total you will be backing up 25 disks instead of 20, which if you value this data will be well worth it. Paul MaudDib fucked around with this message at 12:16 on Feb 8, 2017 |
|
#
?
Feb 8, 2017 12:13
|
|
|
Otaku Alpha Male posted:I'm thinking about buying a Synology DS115J to use with my existing hard drive. Otherwise it's an ok-ish, if modest, plan to go from practically no backup to having some protection against single drive failure. Provided you automate it this time. I do the same thing, with another periodical duplicate on a drive in a usb enclosure and some key documents uploaded to a free onedrive account. As long as you don't delude yourself about what it protects you against.
|
|
#
?
Feb 8, 2017 13:31
|
|
|
Glacier pricing is super-cheap if your data set is reasonably small. 1TB costs 40c/mo, and bulk retrieval (5-12hrs response) for it would cost $2.50. Even Standard retrieval would only be $10/TB. So if that's about all you have...it'd almost be dumb not to throw a copy up there, assuming you have the upload to do so with.
|
|
#
?
Feb 8, 2017 14:37
|
|
|
DrDork posted:Glacier pricing is super-cheap if your data set is reasonably small. 1TB costs 40c/mo, and bulk retrieval (5-12hrs response) for it would cost $2.50. Even Standard retrieval would only be $10/TB. So if that's about all you have...it'd almost be dumb not to throw a copy up there, assuming you have the upload to do so with. Well, I still think that's an underestimation, but 1 TB should be something like $30 in retrieval costs if you need it and you are willing to wait. http://liangzan.net/aws-glacier-calculator/
|
|
#
?
Feb 8, 2017 14:43
|
|
|
Yeah, "willing to wait" being the operative clause. My assumption was that the 1TB of data could be slowly retrieved over a period of several days, vice needing to be all downloaded as fast as possible, which, indeed, would jack prices way the gently caress up. CrashPlan and other online storage solutions are therefore probably a better option should you need the ability to recover a large amount of data rapidly, since they do not charge for retrieval. And at $60/yr, their overall prices are pretty low, too.
|
|
#
?
Feb 8, 2017 14:58
|
|
|
Thank you for mentioning Amazon Glacier, I wasn't really aware of Amazons cloud storage services. I'm still trying to wrap my head around their pricing using their FAQ / third party calculators. I'm not sure I get it. Let's say I want to backup 1TB and this backup will be refreshed once a week. I'm assuming this can be automated so that only new or edited files will be added/replaced in the backup. So for this example, 10gb of that cloud backup will be deleted & replaced every week. Normally, none of the data would have to be retrieved. This is what I came up with:  Edit: I will also take a look at crashplan, thanks for the suggestion. Although I believe I would be fine with slow data retrieval.
|
|
#
?
Feb 8, 2017 15:20
|
|
|
The problem with Glacier is that storage is cheap and retrieval is expensive. It's cheap as long as you never have to pull back data. If you do, the longer the period you can stretch it over, the better. Check out something like Amazon Jungle Disk.
|
|
#
?
Feb 8, 2017 15:27
|
|
|
DrDork posted:Glacier pricing is super-cheap if your data set is reasonably small. 1TB costs 40c/mo, and bulk retrieval (5-12hrs response) for it would cost $2.50. Even Standard retrieval would only be $10/TB. So if that's about all you have...it'd almost be dumb not to throw a copy up there, assuming you have the upload to do so with. Paul MaudDib posted:Well, I still think that's an underestimation, but 1 TB should be something like $30 in retrieval costs if you need it and you are willing to wait. When I use the calculator, it tells me that rertieving 1000gb with a 12h response would actually cost around 800$ depending on my location. What am I not getting here?
|
|
#
?
Feb 8, 2017 15:47
|
|
|
Paul MaudDib posted:1 TB is still within the amount that you can reasonably back up by yourself. Burn 25 GB BD-R discs, with 20 GB of data and a 5 GB parity file from the last disk (via DVDisaster). That way if you get multiple corrupted disks in a row you can use disk N to recover disk N-1, and all the way on back to the start of the corruption. In total you will be backing up 25 disks instead of 20, which if you value this data will be well worth it.
|
|
#
?
Feb 8, 2017 15:52
|
|
|
Otaku Alpha Male posted:When I use the calculator, it tells me that rertieving 1000gb with a 12h response would actually cost around 800$ depending on my location. 12h is a very fast retrieval as far as Glacier is concerned. At 12 hours retrieval time, as far as the actual "retrieval" part of the bill is concerned - that costs you $600. Retrieve the same data set over 256 hours and it costs you $29. Plus extra storage time I guess. Combat Pretzel posted:Question is whether the cost and also time investment of these BD-R backups is actually worth it the added availability of the backups. For the monetary cost of doing 10 backups, you can actually get 4 WD Reds of a terabyte each, to have a set tp rotate. While you might potentially be a backup behind, should one disk break, you don't have the hassle of changing 25 discs and growing a grey beard, plus it's a quasi one-time cost (until the first of the disks eventually breaks, which would however be a long time) while BD-Rs keep costing you. For how much data? BD-R cost like $0.50 per disk (I honestly remember it being half that). That means the whole set of $25 costs $12 amortized. Sure if you spend $400 on a set of 4 WD Reds it's more reliable. On the other hand you could burn 32 sets of each of the BD-Rs instead, and that includes parity protection. So for you to lose your data, 32 copies of the same disc would have to be destroyed. Yes, if you need to randomly access huge quantities of data then BD-Rs are not for you*, that's implicit in the whole "25 GB per disc, 20 GB after parity" thing. The point here is a backup system, not a storage system. How often do you need these discs? Hard drives would be better if random on-line storage were the topic in question. Horses for courses, though, that's not how you do backups. HDDs have never been a cost-effective medium at the scales we're talking about. *: the extra super funny thing here is that Amazon Glacier is a in-house "tape library" that uses BD-XL quad-layer optical media as a storage medium, so the caveat here is it's not for you unless you can commission your own media and media-handling robo-library systems Paul MaudDib fucked around with this message at 16:11 on Feb 8, 2017 |
|
#
?
Feb 8, 2017 16:02
|
|
|
Otaku Alpha Male posted:When I use the calculator, it tells me that rertieving 1000gb with a 12h response would actually cost around 800$ depending on my location. They do a poor job of explaining how retrieval costs work. That calculator also appears to be making a lot of assumptions about how your files are stored. The larger the number of files (regardless of size) the more expensive recovery is. Hence it is better (from a cost perspective) to upload a single 1GB zip of photos rather than 1000 1MB individual photos, though of course this complicates iterative backups somewhat. Amazon itself explains it like this: Amazon Glacier posted:Q: How much do Bulk retrievals cost? Bulk has a 5-12 hour "prep time," so effectively you make the retrieval request, wait 5-12 hours, and then it sends you a note saying it's ready for you to start retrieving. Shorter "prep times" increase the costs. But this isn't the entire story. As Paul notes, a lot of the cost ends up being calculated by figuring our your "peak transfer" for a month, and then retroactively applying that rate to the entire month. So the slower you get your data, the cheaper it is. Unfortunately, I don't think there's any obvious or up-front mention of how this all plays out in any of Glacier's tools, so people occasionally end up with massive bills for doing what they figured should have been cheap and pedestrian operations.
|
|
#
?
Feb 8, 2017 16:19
|
|
|
DrDork posted:Bulk has a 5-12 hour "prep time," so effectively you make the retrieval request, wait 5-12 hours, and then it sends you a note saying it's ready for you to start retrieving. Shorter "prep times" increase the costs. Thanks, this is what I wanted to get at here. Glacier is good in theory but there are some hidden costs to be aware of if you actually need to pull it back. If so - don't do it fast or you will pay like 10-100x as much. Paul MaudDib fucked around with this message at 17:04 on Feb 8, 2017 |
|
#
?
Feb 8, 2017 17:02
|
|
|
Paul MaudDib posted:Thanks, this is what I wanted to get at here. Glacier is good in theory but there are some hidden costs to be aware of if you actually need to pull it back. If so - don't do it fast or you will pay like 10-100x as much. Google Coldline storage seems to be a lot more predictable and less punitive if you need to actually restore a backup. Glacier has so many gotchas that it keeps me away.
|
|
#
?
Feb 8, 2017 17:13
|
|
|
Blackblaze B2 seems pretty reasonable, too and pretty straightforward.
|
|
#
?
Feb 8, 2017 17:14
|
|
|
Thanks for clearing that up! One thing I forgot: I did a speed test and my upload speed is only around 1.5 - 2mbps. Is that feasible?
|
|
#
?
Feb 8, 2017 17:27
|
|
|
Depends on your definition of feasible and how much new data you'd be looking to push each week. For reference: At 2Mbps, 1GB would take 66.67 minutes to upload. 1TB would therefore take about 1140 hours, or the better part of 48 days of solid uploading. If you only need to do that once, while annoying, it's doable. If you're trying to upload 10's of gigs a week, you'd quickly outrun your ability to push it over to the cloud. Also, 2Mbps? That's hideous, and I feel sorry for you having to deal with that.
|
|
#
?
Feb 8, 2017 18:10
|
|
|
Paul MaudDib posted:For how much data? Paul MaudDib posted:BD-R cost like $0.50 per disk (I honestly remember it being half that). That means the whole set of $25 costs $12 amortized. Sure if you spend $400 on a set of 4 WD Reds it's more reliable. On the other hand you could burn 32 sets of each of the BD-Rs instead, and that includes parity protection. So for you to lose your data, 32 copies of the same disc would have to be destroyed. I'd rather eat the additional upfront cost and lose the questionable advantage of an extensive amount of different snapshots in time, over the hassle of burning 25 BD-Rs, especially in regards to time needed. Four disks in a rotation should be sufficient to prevent loss of all data. BD-Rs are interesting for shelfing static media, like sets of photography. But if I'm doing that, I'm doing that relatively soon after the moment the data has been pulled from the source device in question (like the camera). But gently caress me, I'd rather die before I'm going to backup a computer or NAS wholesales on BD-Rs.
|
|
#
?
Feb 8, 2017 19:04
|
|
|
DrDork posted:Also, 2Mbps? That's hideous, and I feel sorry for you having to deal with that.  I'm good for cloud backups, right?
|
|
#
?
Feb 9, 2017 00:52
|
|
|
Grog posted:Uhhhhhhhhh I think maybe 2 or 3 more carrier pigeons and you're good to go
|
|
#
?
Feb 9, 2017 01:20
|
|
|
Seriously, at that point you might as well just stuff an external HDD under a rock down by the river as your "off site backup."
|
|
#
?
Feb 9, 2017 02:58
|
|
|
Otaku Alpha Male posted:Thanks for clearing that up! I use google all the time to figure out download speeds 1 TB / 1.5 Mbs = 61.7283951 days
|
|
#
?
Feb 9, 2017 04:31
|
|
|
DrDork posted:Seriously, at that point you might as well just stuff an external HDD under a rock down by the river as your "off site backup." Isn't hysteresis enough of a problem to warrant saving your data long-term in microfiche instead?
|
|
#
?
Feb 9, 2017 05:34
|
|
|
Greatest Living Man posted:Isn't hysteresis enough of a problem to warrant saving your data long-term in microfiche instead? Do you have a burning need to hide your data under a rock for a century? Generally, it's considered acceptable to copy the data every few years, making hysteresis a non-issue.
|
|
#
?
Feb 9, 2017 05:40
|
|
|
Grog posted:Uhhhhhhhhh Hey look you're Aussie now. (You're actually better off than me and this is why cloud solutions don't exist for me)
|
|
#
?
Feb 9, 2017 13:23
|
|
|
Platystemon posted:Do you have a burning need to hide your data under a rock for a century? What's the current accepted way to include digital data in a time-capsule?
|
|
#
?
Feb 9, 2017 16:05
|
|
|
Greatest Living Man posted:What's the current accepted way to include digital data in a time-capsule? QR-codes lithographically etched in quartz/silicon dies, with a magnifier and QR code decoding stick etched on stainless steel is one I've heard bandied around. Basically super high tech microfiche.
|
|
#
?
Feb 9, 2017 17:00
|
|
|
Methylethylaldehyde posted:QR-codes lithographically etched in quartz/silicon dies, with a magnifier and QR code decoding stick etched on stainless steel is one I've heard bandied around. Basically super high tech microfiche. It's more like a modern cuneiform stone tablet.
|
|
#
?
Feb 10, 2017 15:52
|
|
|
I'm new to the NAS world and working on building my first system (technically still deciding what I want to do). Plan on setting up a server with ESXi and running multiple VMs, including a NAS. Been looking a lot at ZFS as this thread and most NAS blogs seem to have a hard-on for it; but it kind of seems like overkill for a home media server. I don't like the inflexibility and general requirements it has, at least from a home use scenario. I was also looking at mergeFS + snapRAID, and to be honest it seems like a much better and robust solution for my needs, I was wondering what experiences y'all have had with them? Here is an interesting article on using them on a media server https://www.linuxserver.io/2016/02/02/the-perfect-media-server-2016/
|
|
#
?
Feb 14, 2017 22:27
|
|
|
While I can't say much about snapRAID (having never used it), but if you're considering a setup with more than 3 drives, ZFS really isn't overkill at all. It is somewhat inflexible as far as expansion goes, but a little forward planning can ease most of that. If you're planning on doing ESXi, make sure whatever hardware you get supports vt-d so you can passthrough as much hardware as possible to whatever OS you do decide on.
|
|
#
?
Feb 14, 2017 23:33
|
|
|
Also check out Unraid if you're looking for non-ZFS solutions and want more flexibility.
|
|
#
?
Feb 14, 2017 23:34
|
|
|
|
| # ? May 15, 2024 17:37 |
|
|
I just noticed that one of my drives in my additional backup unit (a Synology DS210+) has a reallocated-sector-count that's very near the threshold, so I've ordered 2x2TB to replace the 2TB drives in it (since it doesn't support more than 2TB according to the specs) - but do any of you happen to have replaced a drive in a 2-unit NAS bay, and can you tell me whether I can hotswap the failed drive, or whether I have to turn it off and replace it? I assume that once I do replace it, rebuilding will start automatically since it's RAID1/SHR1.
|
|
#
?
Feb 16, 2017 11:49
|
|
