|
Arivia posted:this got me realizing i really, really need to set up backups myself at home. The best value for money is usually the 8TB or larger externals when they're on sale. Those are also not the territory of SMR (shingled magnetic recording) which can be slower, yet. With disks under 8TB you're more likely to get an SMR disk inside. There are regular sales on the large externals although using a site like https://shucks.top/ that compiles their pricing is great, it seems to only poll US sellers. It's a good idea to setup automated backup, there's a variety of software to do it. I use Cobian backup but it's old and a little janky. I'm sure there's some other stuff the thread could recommend. The main downsides to only having a single disk always attached is that you're adding redundancy but not protected against filesystem attacking software like cryptolocker stuff and physical damage like if there was a fire. For the former you could look at a two bay NAS if it's in your budget, but for the latter the main option these days is an online backup service. That all said, any backup is better than no backup since drives will just die and it's super important to back up anything you don't want to lose.
|
#
?
Jan 30, 2022 02:23
|
|
|
|
| # ? May 23, 2024 10:15 |
|
|
Rexxed posted:The best value for money is usually the 8TB or larger externals when they're on sale. Those are also not the territory of SMR (shingled magnetic recording) which can be slower, yet. With disks under 8TB you're more likely to get an SMR disk inside. There are regular sales on the large externals although using a site like https://shucks.top/ that compiles their pricing is great, it seems to only poll US sellers. I just genuinely don't have any use for 8TB of space. That 1.5 full of a 3TB drive is like five years of pack rat stuff for me, so I'm not concerned that my storage needs are going to super explode out of nowhere. I realized I have a spare (actually two) m.2 slots in my computer so I picked up a 1TB Samsung 970 for that need, unless anyone disagrees I'll pick up that WD in 5TB soon.
|
|
#
?
Jan 30, 2022 02:29
|
|
|
Rescue Toaster posted:Apparently LTT had some problems with their multiple 1.2PB arrays for storing all their footage: I watched the video earlier and I feel like he mentioned that there was an auto-shutdown solution. I know he didn't mention a UPS though. This is making me realize that I should really have a proper mirrored backup. Is that 5 disk expansion thing Synology makes worth it? I've only seen it for like $600 which seems like a really bad deal. Is there a cheaper way to expand the storage for a DS920+ without buying synology's expansion or buying a second NAS?
|
|
#
?
Jan 30, 2022 03:06
|
|
|
Arivia posted:
It's extremely funny to me that there's a class of "tech" YouTubers, in comparison to whom Linus seems like a professional. Like what the heck do those other people do
|
|
#
?
Jan 30, 2022 03:40
|
|
|
Arivia posted:this got me realizing i really, really need to set up backups myself at home. I have a couple of the Seagate 5TB 2.5" drives I burned in with badblocks and was going to shuck and put into an icy dock and rotate, but now I'm thinking there's not much advantage to doing that over just using them on USB 3.0, since smart works fine and everything. Right now I'm trying to decide between trying to get things setup with the XigmaNAS setup I have which is pretty janky, switch to a new NAS distro, or do my own setup on linux. Ages ago I did setup a nas with mdadm & samba and everything myself on linux, but I'd want to find some kind of easy guide especially to setting up and automating ZFS stuff on linux if I was going to go that route again.
|
|
#
?
Jan 30, 2022 03:44
|
|
|
CerealKilla420 posted:I watched the video earlier and I feel like he mentioned that there was an auto-shutdown solution. I know he didn't mention a UPS though. They had a UPS but it lit on fire.
|
|
#
?
Jan 30, 2022 03:47
|
|
|
To be fair they did replace their UPS, so they did have one, they just didn't configure it properly.
|
|
#
?
Jan 30, 2022 03:59
|
|
|
Buff Hardback posted:To be fair they did replace their UPS, so they did have one, they just didn't configure it properly. If they have a big network UPS, at least last I tried it FreeNAS would mis-configure NUT so that it would tell the NUT server it was ready for power down before even attempting shutdown. Oops!
|
|
#
?
Jan 30, 2022 04:35
|
|
|
Combat Pretzel posted:Bitrot due to powerloss shutdowns? What is he talking about? Isn't this backwards metadata tree writes with uberblock being last supposed to avoid this? What can happen during powerloss is that you lose transaction groups that're in flight, but because ZFS is atomic that data hasn't been written to disk yet so ZFS will throw away partial txgs when the pool is next imported. No, having L2ARC is fine if your resident memory set is bigger than the maximum memory that your system can work with. Having two of them when they're SATA disks is dumb because you don't get multiple queues like you do with NVMe, the bandwidth of the interface is slower, and the access latency is many orders of magnitude slower. Klyith posted:I dunno how power loss could be causing that quantity of errors to a storage system that is almost entirely data at rest. This is their dump of archival footage, by their own description they barely touch the thing. A power loss shouldn't cause widespread "bitrot" to old, static data. I don't know the details since I don't make a habit of watching LTT, but he worked with some other YouTubers where he built them storage servers. Fame Douglas posted:LTT is a vapid entertainment channel, don't take what they do so seriously. Not an hour after I posted before going to bed, someone on IRC made a remark to the effect that "ZFS clearly isn't as good as everyone says" while linking to the video.
|
|
#
?
Jan 30, 2022 07:13
|
|
|
BlankSystemDaemon posted:Not an hour after I posted before going to bed, someone on IRC made a remark to the effect that "ZFS clearly isn't as good as everyone says" while linking to the video.
|
|
#
?
Jan 30, 2022 11:45
|
|
|
Combat Pretzel posted:Probably a BTRFS user.
|
|
#
?
Jan 30, 2022 11:49
|
|
|
All this talk made me have a closer look at my brand new zfsonlinux server Jan 28 23:34:58 kernel: [1424496.909541] mce: [Hardware Error]: Machine check events logged Jan 28 23:34:58 kernel: [1424496.909545] [Hardware Error]: Corrected error, no action required. Jan 28 23:34:58 kernel: [1424496.909551] [Hardware Error]: CPU:1 (19:21:0) MC7_STATUS[Over|CE|-|-|-|-|-|-|-]: 0xd00a000000000000 Jan 28 23:34:58 kernel: [1424496.909555] [Hardware Error]: IPID: 0x0000000000000000 Jan 28 23:34:58 kernel: [1424496.909557] [Hardware Error]: L3 Cache Ext. Error Code: 0, Shadow Tag Macro ECC Error. Jan 28 23:34:58 kernel: [1424496.909560] [Hardware Error]: cache level: RESV, tx: INSN google suggests the 5.08 kernel that ubuntu 20.04 defaults to is too old for ryzen 5000 cpus. Some more digging and it turns out you can upgrade with apt install --install-recommends linux-generic-hwe-20.04 to get kernel 5.13 instead. Hopefully it won't spit out more scary cryptic messages.
|
|
#
?
Jan 30, 2022 12:03
|
|
|
Machine check exception due to ECC correcting an error in L3 is probably an indication that the CPU ain't doing so well. Also, there's no such thing as ZFS on Linux; there's just OpenZFS, Illumos ZFS and descendants (FreeBSD 12, NetBSD, macOS and Windows ports), and OracleZFS. There's work ongoing to move all the non-OracleZFS implementations to OpenZFS. BlankSystemDaemon fucked around with this message at 12:37 on Jan 30, 2022 |
|
#
?
Jan 30, 2022 12:30
|
|
|
Arivia posted:are there any problems with the above devices/recommendations for windows 10 backup software? ideally something plug and play like Apple's Time Machine would be great, just leave the external HD plugged in and have it back itself up every so often. I'm not opposed to a different model of external HD, even if it's a big chonker that takes a separate power brick or something, I just grabbed the WD elements since the thread seems big on those. File History does pretty much exactly what you describe. I've had good luck with it, but I haven't had to recover too much with it: https://www.howtogeek.com/74623/how-to-use-the-new-file-history-feature-in-windows-8/ nitsuga fucked around with this message at 15:19 on Jan 30, 2022 |
|
#
?
Jan 30, 2022 14:53
|
|
|
Well a drive in my 8 drive raidz2 just started throwing errors. I purchased them (a mix of 2tb drives at... $80 a pop) a month before the Thailand floods, they've all had a good life. Just bought a (larger) replacement, it looks like drive prices have decreased, but not at the rate they did from 2000-2010. I can still buy a 2TB drive for 80 bucks.
|
|
#
?
Jan 30, 2022 15:20
|
|
|
I'm still trying to decide whether I want a third spindle in my mirror or not, since I'm hosting Steam games on it, too. You know, IOPS for cold data.
|
|
#
?
Jan 30, 2022 16:27
|
|
|
nitsuga posted:File History does pretty much exactly what you describe. I've had good luck with it, but I haven't had to recover too much with it: https://www.howtogeek.com/74623/how-to-use-the-new-file-history-feature-in-windows-8/ It is exceptionally useful for a tiny usecase, but when you hit that usecase it's always pleasant that it's possible. alo posted:Well a drive in my 8 drive raidz2 just started throwing errors. I purchased them (a mix of 2tb drives at... $80 a pop) a month before the Thailand floods, they've all had a good life. Combat Pretzel posted:I'm still trying to decide whether I want a third spindle in my mirror or not, since I'm hosting Steam games on it, too. You know, IOPS for cold data.
|
|
#
?
Jan 30, 2022 19:26
|
|
|
Arivia posted:they've done a bunch of videos where they make a NAS server for iJustine or similar other youtubers. Then the youtubers go on to buy a different NAS solution because it didn't suit their needs.
|
|
#
?
Jan 30, 2022 19:51
|
|
|
BlankSystemDaemon posted:Yeah, there's a strange phenomenon where the price per TB is anything but linear despite the fact that you'd think the smaller drives wouldn't be selling at all with the much bigger drives as relatively cheap as they are. I'll be replacing each of these with 14tb drives. I don't expect to get >10y of power on time again, but we'll see. Looks like I'll be paying between 250 and 320 dollars per drive now. Also thinking about replacing the enclosure with something a bit more robust (moving the drives to a sas/sata jbod sounds nice). Back when I put the system together, the Norco 4224 was the popular enclosure. It looks like in the last 10 years there's been a lot more old jbods appearing in the used market that might be better suited for home storage (or at least not suck like the Norco's). I don't need a netapp disk shelf sized jbod, but is there anything in the 12-16 drive range that's not terrible? Sound isn't an issue since this is in a dedicated room in my basement (in a nice and cozy rack).
|
|
#
?
Jan 30, 2022 20:29
|
|
|
BlankSystemDaemon posted:Yeah, there's a strange phenomenon where the price per TB is anything but linear despite the fact that you'd think the smaller drives wouldn't be selling at all with the much bigger drives as relatively cheap as they are. In years past I've said it's because there is a certain minimum value to having "a drive" for systems where you don't actually care how much storage it has. At this point I would have thought the existence of cheap-as-gently caress SATA and NVMe SSDs in the 500GB-1TB range would have broken that, but apparently there's still enough demand specifically for cheapish small spindles to justify their production and existence. alo posted:I don't need a netapp disk shelf sized jbod, but is there anything in the 12-16 drive range that's not terrible? Sound isn't an issue since this is in a dedicated room in my basement (in a nice and cozy rack). Xyratex is the original manufacturer for the Netapp DS4243/4246 shelves, and they also made smaller 12-bay shelves branded for Dell/Compellent and others. IOwnCalculus fucked around with this message at 20:43 on Jan 30, 2022 |
|
#
?
Jan 30, 2022 20:38
|
|
|
Rescue Toaster posted:I have a couple of the Seagate 5TB 2.5" drives I burned in with badblocks and was going to shuck and put into an icy dock and rotate, but now I'm thinking there's not much advantage to doing that over just using them on USB 3.0, since smart works fine and everything. So you're using an external enclosure connected to the NAS via USB? I have 3 2tb 3.5in drives laying around - I might just put them to use to backup my essential data.
|
|
#
?
Jan 30, 2022 20:41
|
|
|
CerealKilla420 posted:So you're using an external enclosure connected to the NAS via USB? I've done some manual backups, if I keep them in the USB I'll need to add a proper PCIe USB 3.0 controller and external hub that I can reach since my motherboard doesn't have one, and the case doesn't have USB 3.0 on the front. That's kind of why I was thinking using the icy dock would be a lot cleaner, don't have to worry about cables and bullshit and adding a tray in the rack to set the drives on while they're hooked up. But I haven't tested hot swapping SATA in FreeBSD (XigmaNAS) yet in the icy dock. I have a lot of poo poo on my NAS that realistically doesn't need to be backed up, that could be re-acquired. So I was thinking a script that does a find for folders containing a .backup hidden file, and generate a list of folders for rsync to use. Nothing more complicated than that is needed probably. Ideally it would automatically mount/unmount to refresh the backup weekly while the drive was attached, and when it's removed and swapped, automatically kick off a new backup immediately. Something like that. Not sure if there's anything like that already or if I need to do it myself, and how hard it would be to do that inside XigmaNas or TrueNAS or if I'm going to go back to a plain ubuntu LTS & openZFS setup next.
|
|
#
?
Jan 30, 2022 20:50
|
|
|
BlankSystemDaemon posted:As in a three-way mirror? I'm not sure that gives you more IOPS.
|
|
#
?
Jan 30, 2022 21:40
|
|
|
BlankSystemDaemon posted:File History, and ZFS snapshots for that matter (which, interestingly, be integrated with Samba to appear like File History in Windows), is just versioned files. Oh, yeah, I gave a misleading example. I'm less concerned about versioning files and more just a plug and play simple automatic backup solution like what Time Machine does. I'd ideally have something I can use to restore full drives instead of Time Machine's infamous instability with the versioning functionality.
|
|
#
?
Jan 30, 2022 21:47
|
|
|
If anybody has rolled your own NAS setup (did your own samba, openzfs, etc...) instead of using a xyzNAS distro, any links to particular configuration guides you found useful? There's so much ancient info online sometimes it can be hard to tell if you're doing things the right way anymore. I think I can work out the basic setup, muddle my way through samba & UPS fine again, but things like getting an email if the weekly scrub fails, or monitoring smart... that kind of stuff is annoying since the NAS distros at least do it but they never quite do it quite right.
|
|
#
?
Jan 30, 2022 22:30
|
|
|
Combat Pretzel posted:The vdev does round-robin scheduling for reads. --ninja edit: Actually, it's slightly more complicated, since it considers locality and stuff like that. Arivia posted:Oh, yeah, I gave a misleading example. I'm less concerned about versioning files and more just a plug and play simple automatic backup solution like what Time Machine does. I'd ideally have something I can use to restore full drives instead of Time Machine's infamous instability with the versioning functionality. Otherwise, there's BareOS although I'm not quite sure it does full system imaging. The real advantage of BareOS is that it supports tape and tape libraries. Rescue Toaster posted:If anybody has rolled your own NAS setup (did your own samba, openzfs, etc...) instead of using a xyzNAS distro, any links to particular configuration guides you found useful? There's so much ancient info online sometimes it can be hard to tell if you're doing things the right way anymore. Periodic can also be used to handle ZFS snapshots, and those snapshots integrate with File History in Windows, as previously mentioned. BlankSystemDaemon fucked around with this message at 22:44 on Jan 30, 2022 |
|
#
?
Jan 30, 2022 22:35
|
|
|
BlankSystemDaemon posted:Sure, but doesn't that only result in the first disk returning the data being the one to win the race, so to speak? --edit: A quick look at vdev_mirror.c, it seems like it evaluates the load of each child in the mirror and selects the one with the least load (based on queue length), plus some selective magic for rotational media. Combat Pretzel fucked around with this message at 23:21 on Jan 30, 2022 |
|
#
?
Jan 30, 2022 23:13
|
|
|
As I understand it vdev mirrors for reads are a variation of smart first past the post (each additional mirror offering roughly on average 50% better latency presuming all things being equal unless there’s multiple banks of vertically aligned heads to work with) while writes require all mirrors to finish writes unless some crazy zfs flag allows partial ack writes and users accept what amounts to the RAID write hole on a file system meant to avoid it. Rotational media it totally makes sense to try to determine which general area of the disk the heads are vs where the data is expected given that the ms order magnitude latency is several off from the sub-us range to compute the prediction and plan head movement across different vdevs.
|
|
#
?
Jan 31, 2022 03:56
|
|
|
It isn't that sophisticated, tho. It just adds a penalty to the load value based on locality. The IO elevator sorting seems to operate independently of that. Either way, I was just wondering about bumping IOPS/lowering latency. The 10TB mirror is just fine so far, for the bits I want to keep and offloading apps and games. A third disk is cheaper than another pair. I don't really know why some people have like 50-100TB NASes for mostly just storing movies, anyway. Then again, this is the packrats thread.
|
|
#
?
Jan 31, 2022 15:17
|
|
|
What is everyone's favorite versioned backup software + platform?
|
|
#
?
Jan 31, 2022 17:34
|
|
|
Smashing Link posted:What is everyone's favorite versioned backup software + platform? There's a dedicated backup thread where a lot of people like Duplicati, but that's because they're backing up to the cloud and Duplicati is very much optimized for that. IMO it's meh for backing up to your own NAS, because it turns everything into encrypted data chunks that aren't readable if you lose your configuration files. I was asking about alternatives for local backup that avoided that issue, and got a pointer to UrBackup. Tried it out and thought it was pretty neat. If I was backing up to a NAS I would be using that, but I backup to plain removable drives and decided it was more than I needed. (Instead I figured out a fun way to get incremental backup + history with batch script & robocopy.)
|
|
#
?
Jan 31, 2022 18:13
|
|
|
Smashing Link posted:What is everyone's favorite versioned backup software + platform? Syncthing and Backblaze
|
|
#
?
Jan 31, 2022 18:26
|
|
|
Klyith posted:There's a dedicated backup thread where a lot of people like Duplicati, but that's because they're backing up to the cloud and Duplicati is very much optimized for that. IMO it's meh for backing up to your own NAS, because it turns everything into encrypted data chunks that aren't readable if you lose your configuration files. I recently set up Duplicati for the first time about a month back, admittedly for backing up to Gdrive as Klyith said, but there was definitely an option to create the chunks unencrypted. Not saying that it's the best solution here, I'd leave that to people more knowledgeable than I am but figured I'd mention it. Scruff McGruff fucked around with this message at 19:55 on Jan 31, 2022 |
|
#
?
Jan 31, 2022 18:26
|
|
|
Scruff McGruff posted:I recently set up Duplicati for the first time about a month back, admittedly for backing up to Gdrive as Klyith said, but there was definitely an option to create the chucks unencrypted. Not saying that it's the best solution here, I'd leave that to people more knowledgeable than I am but figured I'd mention it. Ah, true -- but the chunked up data format is AFAIK non-optional and that's the part that is near-impossible to recover from if you lose your config files. The encryption is just a password. Duplicati is doing true incremental with compression and stored changes, which reduces bandwidth + storage by a ton (important when you're paying for both). That data format was what made me feel uncomfortable with it for local backup, where those factors don't apply. UrBackup was much more appealing because the current most-recent backup is just straight-forward copies of your files, easy peasy. However I am not at all experienced with Duplicati and this is just stuff I've seen from using the program exactly once, looking at the docs, and perusing their forums. Also it was 3 years ago when I last looked and it's a project that has very active development. So I could be totally wrong!
|
|
#
?
Jan 31, 2022 19:53
|
|
|
Thanks for the replies and pointer to the Backup thread. Right now I am playing around with Duplicati and Syncthing and considering giving Borg a trial. So far with Duplicati I have about 0.5 TB synced to Gsuite and it hasn't choked, so that's a good thing I guess. Going to start pushing more data onto Syncthing to see how it performs next.
|
|
#
?
Jan 31, 2022 20:07
|
|
|
I used Duplicati to backup my home server to my other home server for a few years, but recently swapped it over for Borg. It tended to get silently broken over time and stop backing up due to a missing file or permission error, requiring manual intervention. Way too unreliable for me, especially after the last time when I realised it hadn't done a complete backup for over three months. So far Borg (via Borgmatic) hasn't had any of the same problems.
|
|
#
?
Jan 31, 2022 22:24
|
|
|
Rescue Toaster posted:If anybody has rolled your own NAS setup (did your own samba, openzfs, etc...) instead of using a xyzNAS distro, any links to particular configuration guides you found useful? There's so much ancient info online sometimes it can be hard to tell if you're doing things the right way anymore. I just found https://github.com/davestephens/ansible-nas the other night and will be giving it a try this time around
|
|
#
?
Feb 1, 2022 17:11
|
|
|
If you've got Samba listening on the web, first of all why? Second of all, you probably shouldn't. BlankSystemDaemon fucked around with this message at 18:02 on Feb 1, 2022 |
|
#
?
Feb 1, 2022 17:53
|
|
|
BlankSystemDaemon posted:If you've got Samba listening on the web, first of all why? I would assume most people won't have the Fruit module enabled and the way I understood the announcement it wouldn't affect them.
|
|
#
?
Feb 1, 2022 18:01
|
|
|
|
| # ? May 23, 2024 10:15 |
|
|
Saukkis posted:I would assume most people won't have the Fruit module enabled and the way I understood the announcement it wouldn't affect them. Also, excuse me while I'm over here, shaking my head at using implementing mdoc/groff in xml.
|
|
#
?
Feb 1, 2022 18:02
|
|