|
Coxswain Balls posted:Got a backup USB drive running now, but the only other drives I have lying around are 8GB which causes the install to fail on 13.0. FreeNAS 11 installed fine, so I went from a fresh install of 11 and did the upgrade chain from 11 ➡ 12 ➡ 13 to match my config file, then upgrade to current. It took me longer than it should have because man does TrueNAS Core not like my USB 3.0 drives/ports for some reason. Whenever I had it on a USB 3.0 port there was some issue or another during the installation or upgrade paths, but when I upgrade using a 2.0 port then power down and move the boot drive to a blue SS port it seems to run just fine and at full speed. It seems to like locking up and failing during the upgrade script that runs on reboot after installing an update.
|
#
?
Aug 13, 2023 12:49
|
|
|
|
| # ? May 28, 2024 23:29 |
|
|
Recently, one of my 4 Toshiba N300 10TB drives decided to kick the bucket. It wasn't apparent since SMART checked out fine but I noticed a huge amount of I/O coming from that drive, and pulling it from the bay immediately fixed it, so i bought a new one. I also decided to redo my Qnap installation, and put all my *arrs and Plex into Docker. It's nice that I can keep a clean install going this way.
|
|
#
?
Aug 14, 2023 15:01
|
|
|
I was able to grab one of those Intel Optane SSDs for a dollar, so now I need to get an interface for it and I get to learn about M.2 stuff. Looking up M.2 cards I'm seeing things like this expansion card with four M.2 slots, which theoretically seems like it could be useful for adding SATA ports, but since I'm limited to PCI Express 3.0 am I going to be bottlenecked anywhere? I think an M.2 card with five SATA 3.0 ports would max out at 3 GBps; since a 16x v3 slot has 15.754 GBps capacity I can theoretically add 15 SATA ports plus that SSD with no issues, yeah? I have no idea if this is a bad idea from a reliability or even a thermal standpoint.
|
|
#
?
Aug 15, 2023 04:23
|
|
|
The adapter card you linked only works for NVMe SSDs, not SATA SSDs, and only works if your motherboard can treat your x16 slot as if it is four x4 slots instead. If it can, then all four PCIe SSDs will get the maximum bandwidth your system can push over an x4 slot.
|
|
#
?
Aug 15, 2023 04:48
|
|
|
IOwnCalculus posted:The adapter card you linked only works for NVMe SSDs, not SATA SSDs, and only works if your motherboard can treat your x16 slot as if it is four x4 slots instead. If it can, then all four PCIe SSDs will get the maximum bandwidth your system can push over an x4 slot. To clarify, I'm talking about something like this in addition to that adapter.
|
|
#
?
Aug 15, 2023 04:58
|
|
|
Coxswain Balls posted:I was able to grab one of those Intel Optane SSDs for a dollar, so now I need to get an interface for it and I get to learn about M.2 stuff. Looking up M.2 cards I'm seeing things like this expansion card with four M.2 slots, which theoretically seems like it could be useful for adding SATA ports, but since I'm limited to PCI Express 3.0 am I going to be bottlenecked anywhere? I think an M.2 card with five SATA 3.0 ports would max out at 3 GBps; since a 16x v3 slot has 15.754 GBps capacity I can theoretically add 15 SATA ports plus that SSD with no issues, yeah? I have no idea if this is a bad idea from a reliability or even a thermal standpoint. What are the M.2 -> SATA adapters? Whatever is plugged into that card would have to be PCIe/NVME. Were you thinking something like this: https://www.amazon.ca/PCIe-Adapter-Internal-Linux-Windows/dp/B09GW3B92M These are PCIe Gen3 x2 so you're getting 2GB/s transfer which is 2.67 6Gbps drives. So you'd hit some bottlenecking if you wanted to use more than 2 drives per m.2 slot. If there are x4 PCIe -> SATA bridges you could double it. So if you want 15 drives you would definitely hit bottlenecks if you had 3 of the m.2 slots used by Gen3 x2 pcie to sata bridges. Note this is all theoretical bottlenecks, it might not be noticeable unless you really start pushing oversubscribed links. Also the motherboard needs to support bifurcation of x16 to x4x4x4x4 which isn't something most consumer boards do but if you have something from supermicro or asrock rack you should be good. Also curious what refclk buffer that card has because the 100MHz clock needs to be fanned out to all the m.2 slots. it looks like one of the parts near the front might be it but the picture has it blurred so you can't check the part #. The other two parts are for controlling the LEDs I'm guessing.
|
|
#
?
Aug 15, 2023 05:02
|
|
|
using that card with those m.2 to 5 sata adaptors is going to be pain. For a bunch of sata drives, your best option is probably to buy something like an LSI 9300-16i.
|
|
#
?
Aug 15, 2023 05:15
|
|
|
I wouldn't trust my data to a hodge podge setup like that. Get an 9300-16i or similar.
|
|
#
?
Aug 15, 2023 05:17
|
|
|
Yeah, thirding the "do not do that" recommendation. That's too many layers of jank even for me, and you're going to spend more than the right solution would cost. 9300-16i would be the right single-card solution, or if you have at least two x8 slots you could do two separate LSI 2308 controllers (9217, H220, etc, check here: https://forums.servethehome.com/index.php?threads/lsi-raid-controller-and-hba-complete-listing-plus-oem-models.599/), though you will want to find later revision versions of the chipset that are PCIe 3.0 and not the early 2.0 version.
|
|
#
?
Aug 15, 2023 05:44
|
|
|
This is all theoretical for future expansion, since I'm still doing alright with my 4x3TB RAIDZ2 pool + 256GB SSD for VMs. I'm using a TS140 ThinkServer, which gives me:
Wild EEPROM posted:using that card with those m.2 to 5 sata adaptors is going to be pain. Wibla posted:I wouldn't trust my data to a hodge podge setup like that. Get an 9300-16i or similar. Good to know, thanks. I know jack poo poo about expansion cards so I don't know if it's a situation where it's fine to use some non-branded part drop-shipped from China, or if an established brand is the way to go. In this case should I get a single slot PCIEv2 M.2 card for slot 3, and save slot 1 for something like the 9300-16i in the future? Or is the chipset in my server so old that it might not even boot from NVMe and I should just get a USB 3.0 enclosure like the one that was linked earlier?
|
|
#
?
Aug 15, 2023 05:44
|
|
|
Yeah those little adapter deelies along with the M.2 expander are pretty janky. I would just go with a proper SAS/SATA controller. I wonder are any SAS expansion boxes showing up on ebay for cheap?
|
|
#
?
Aug 15, 2023 06:17
|
|
|
That server is indeed old enough that it's very unlikely to boot directly from NVMe. If I were using that box and wanted to get my boot SSD out of the physical drive bays, I'd stick a SATA m.2 on one of these and put it in that mostly-useless x1 slot. (If such a thing existed to use the legacy PCI slot instead, I'd do that, but good luck finding one.) It's also worth noting that the BIOS on that box probably doesn't support PCIe bifurcation, so you wouldn't have actually been able to use more than the first slot on that 4x m.2-to-x16 adapter. At any rate, this would leave you with the x4 slot and the x16 slot for further drive expansion. How many drives are you trying to connect in here? Unless you're dealing with a large SAS enclosure on top of this, I don't think you're going to deal with enough spindles to actually saturate even the PCIe 2.0 x4 slot so you would be fine with a cheap LSI 2008 adapter there.
|
|
#
?
Aug 15, 2023 06:23
|
|
|
I'm going to have to upgrade the case and power supply long before I can get additional drives, but it's nice to know what my options are and to keep them open for when the time comes. No point in getting something for my boot drive to work with slot 1 when I can get away with using a less valuable one. I found this post on Reddit about someone whose NVMe drives show up but aren't bootable, except for one drive which works. His best guess is that it works because it's an old-rear end NVMe 1.1 drive, which is the same as the dollar Optane I bought so I'll just need to get my hands on an interface card I can return and give it a shot.
|
|
#
?
Aug 15, 2023 07:23
|
|
|
How's the reliability on these kinds of external storage drives? The price seems pretty good for the amount of storage? I had bought some flash drives that were 2 TB but they seem to fail on me and the files were unreadable after copying them.  https://www.amazon.ca/Western-Digital-Elements-Desktop-Drive/dp/B07YD3G568/ref
|
|
#
?
Aug 16, 2023 02:11
|
|
|
Those WD Elements are just a regular 3.5" drive inside an external enclosure with a USB-SATA interface. As long as you don't use it in place of a football it should be as reliable as any other hard drive. In the (rare and unlikely) event that the failure of the drive is the result of the USB-SATA interface, you can take the drive out and use any other SATA interface to read it. A lot of us in this thread buy or have bought those drives specifically because they're cheaper than the same drive sold as an internal drive, even though we just rip them out of the enclosure ASAP. Finally - if you bought suspiciously-cheap large flash devices on Amazon or Walmart or basically any large online retailer, there's a good chance they were part of the current crop of blatant fake drives. Unscrupulous sellers will configure the drive to show up as if it is 2TB or 16TB or whatever they're advertising it as, but if you try to actually write that much data to it, it will fail in one of a handful of data-destructive ways.
|
|
#
?
Aug 16, 2023 05:03
|
|
|
I shucked a bunch of 14TB WD elements drives that I bought cheap used from a chia farmer, they've held up better than I expected.
|
|
#
?
Aug 16, 2023 05:58
|
|
|
Though note that some of the 2.5" drives have the USB controller straight on the drive and no SATA in sight. I don't think that will ever make sense for 3.5" drives, though.
|
|
#
?
Aug 16, 2023 10:11
|
|
|
Does anyone know if newegg ever puts this Rosewill 4u 12 hotswap bay case on sale? https://www.newegg.com/rosewill-rsv-l4412u-black/p/N82E16811147330
|
|
#
?
Aug 16, 2023 15:36
|
|
|
Computer viking posted:Though note that some of the 2.5" drives have the USB controller straight on the drive and no SATA in sight. I don't think that will ever make sense for 3.5" drives, though. True, but I've got to imagine the market for external 2.5" spindles is drying up fast now that 1-4TB flash devices are as cheap as they are.
|
|
#
?
Aug 16, 2023 15:37
|
|
|
phosdex posted:Does anyone know if newegg ever puts this Rosewill 4u 12 hotswap bay case on sale? https://www.newegg.com/rosewill-rsv-l4412u-black/p/N82E16811147330 Man I used to have an older version of that (eventually passed it on for being too loud) and I swear I paid like $120 or something, that price is insane. For that much you should be considering looking up used supermicro stuff on ebay, it'll probably come with a backplane and from a brief look some of them are even cheaper than that thing.
|
|
#
?
Aug 16, 2023 15:40
|
|
|
power crystals posted:Man I used to have an older version of that (eventually passed it on for being too loud) and I swear I paid like $120 or something, that price is insane. For that much you should be considering looking up used supermicro stuff on ebay, it'll probably come with a backplane and from a brief look some of them are even cheaper than that thing. This. You used to be able to find that thing on sale around $175-$200 but I haven't seen that happen in years. I still have an old eBay alert for that case too and even there it's almost always $300 and up. Better off prowling FB Marketplace/Craigslist for local Supermicro, Norco, or Chenbro chassis someone's selling. Ebay works too but shipping for cases can get kind of silly. I feel like Covid really messed up the NAS/Homelab market (in the US at least), it's really difficult to find a case with more than 4 hot-swap bays for under $200 anymore. Scruff McGruff fucked around with this message at 17:07 on Aug 16, 2023 |
|
#
?
Aug 16, 2023 17:02
|
|
|
Yeah I remembered them being cheaper years ago, so this price is tough to swallow.
|
|
#
?
Aug 16, 2023 18:25
|
|
|
I'm trying to spec a system for Plex / *arr / photo and document backup, plus it would replace my RPi for pi-hole and maybe run some home automation in the future. I only need to serve Plex to the local network, and I was thinking of running Unraid. There are a few things I'm unsure on. - Do I want ECC RAM? If so it seems like I'm pretty restricted on motherboard and processor choice, are there any standard recommended combinations? - Unraid supports ZFS now, is that something I want to use? - Any good reason to go with DDR5?
|
|
#
?
Aug 16, 2023 18:43
|
|
|
You don't need ECC but it's nice to have. Full stop. Anyone who says otherwise is going on a bit too hard about data integrity for Linux isos. E: to answer the other questions, don't use ZFS for the main array on UnRAID or you lose the benefit of UnRAID disk flexibility. It's fine for caches though. No need for DDR5 either. Matt Zerella fucked around with this message at 18:57 on Aug 16, 2023 |
|
#
?
Aug 16, 2023 18:52
|
|
|
One of the ZFS devs is on record (might be on the mailing list? Can't remember now honestly) saying that ZFS doesn't benefit more from ECC or that ECC is more necessary than other software RAID systems and went into each of the different transactions in great detail comparing to various other software RAID systems and it made sense. ECC in some cases can mean more crashes / hangs because it means that an unmaskable interrupt (UMI) can be fired off and the kernel can't recover the memory transaction successfully and thus panic). The idea is that instead of making an incorrect write it is preferable to straight up crash / halt the system. With how DDR5 works with a bit better error correction and detection semantics per JEDEC standards things are going to be a bit better for standard DIMMs compared to DDR4. I had a system I didn't realize was having so many RAM errors until I checked the BIOS events and saw a stream of corrected 1-bit errors and replaced the UDIMM. That's part of what ECC buys you in a professional setting as well - more metrics and data to swap out bad hardware. Think of it similar to having BMCs and other management systems embedded into hardware.
|
|
#
?
Aug 16, 2023 20:48
|
|
|
I'm a bit ambivalent about the on-die ECC. One reason it exists is to deal with the decreasing manufacturing tolerances that come from miniaturisation and frequency increases, and increased error rate thereof. IIRC there's no side channel for the chips on the DIMMs to report errors to the OS. That would have been desirable. On the other hand, the manufacturers expect errors, and generic users seeing non-zero counters would probably create a support burden (see these barely decipherable SMART values on drives).
|
|
#
?
Aug 16, 2023 23:19
|
|
|
big scary monsters posted:- Do I want ECC RAM? If so it seems like I'm pretty restricted on motherboard and processor choice, are there any standard recommended combinations? From what I've seen a lot of ASRock AM4 boards claim to support ECC (check the specs page) with Ryzen Pro and non-APU consumer Ryzen chips. I'm currently running it on my TrueNAS box which has an X470 Taichi and 5900X, and it seems to work fine. ASUS has some AM4 ECC boards but it's not as common, and I don't know if MSI or Gigabyte support it at all. If you want Intel you're pretty much locked into server platforms. E3 Xeons in eBayed Supermicro boards are a valid choice, and cheap if you're willing to go back several generations.
|
|
#
?
Aug 16, 2023 23:37
|
|
|
Ryzen ecc support seems full of inconsistencies. Seems like: - if it encounters a problem it will lock up - if it encounters a problem it will just alert but continue - it’s in the cpu but its not officially supported - some manufacturers say “supported” but what they mean is it will run ecc dimms with ecc off What is even going on? Is it officially supported on ryzen pro or something? Is it officially supported on epyc only?
|
|
#
?
Aug 16, 2023 23:51
|
|
|
The ultimate answer there is that not enough people will buy it specifically because it has proper ECC support, for the manufacturers to justify really supporting it. Business users that must have ECC will gladly pay the premium for a fully validated platform; the majority of consumers will not. It's not how things should be, but it's the way they are. It helps that even 10-year-old server-grade hardware has far more CPU than even most of us in here need, and can be bought for less than a minimal new-build of non-ECC consumer-grade hardware. The only place it's deficient is power usage... which sucks.
|
|
#
?
Aug 16, 2023 23:57
|
|
|
Thanks for the replies and discussion! I was planning to go with an Intel chip, guess I'll see what is available at a good price/spec and not worry too much about ECC if it proves annoying to get hold of. I'm in Norway so the market for used stuff is often not so great (or very expensive).
|
|
#
?
Aug 17, 2023 01:08
|
|
|
IOwnCalculus posted:Those WD Elements are just a regular 3.5" drive inside an external enclosure with a USB-SATA interface. As long as you don't use it in place of a football it should be as reliable as any other hard drive. In the (rare and unlikely) event that the failure of the drive is the result of the USB-SATA interface, you can take the drive out and use any other SATA interface to read it. Wibla posted:I shucked a bunch of 14TB WD elements drives that I bought cheap used from a chia farmer, they've held up better than I expected. Computer viking posted:Though note that some of the 2.5" drives have the USB controller straight on the drive and no SATA in sight. I don't think that will ever make sense for 3.5" drives, though. Thanks y'all; so if I want to play it safe, and go for a more economical option, I should aim for the 3.5" external drives? I see a 4ishTB external 3.5" enclosure+drive so a pretty affordable 140$ish (5 TB for like 10$ more), should I spring for this if I got like around ~3 TB worth of data I want to backup and don't want a subscription to an online service? I figure I could spring for this now and then work my way towards a NAS further down the line?
|
|
#
?
Aug 17, 2023 02:48
|
|
|
oof somebody needs to build https://shucks.top for canada
|
|
#
?
Aug 17, 2023 14:51
|
|
|
shame on an IGA posted:oof somebody needs to build https://shucks.top for canada This is somewhat similar: https://diskprices.com/?locale=ca
|
|
#
?
Aug 17, 2023 15:11
|
|
|
Wild EEPROM posted:Ryzen ecc support seems full of inconsistencies. I think it's only officially supported on Epyc and Threadripper, but unlike Intel AMD hasn't bothered to cut the support out on the consumer chips so it's kind of a "eh, it's probably fine but if we did any validation we're not saying and we give zero guarantees" situation. (e: Except the non-Pro APUs. I'm pretty sure it doesn't work on those at all.) There's clearly a certain amount of confidence in it from the board partners though, given that ASRock Rack sells straight up AM4 server boards which use the same chipsets and CPUs. I don't know what other boards do, especially non-ASRock boards, but the first thing I did after getting my NAS together was run Memtest as a basic stability check. I pretty quickly realized one stick in my dual-channel kit was bad because it was throwing consistent errors at the same addresses, but Memtest reported that they were being corrected by ECC and continued as normal. Of course, Memtest doesn't lock up for uncorrectable errors either but it seems like a good indication that what happens after a correctable error is up to the OS. I didn't bother trying to do anything more before returning the kit and as far as I know the replacement has been fine, so I don't know what TrueNAS does on this system if it encounters a correctable error. e: ASUS sells a workstation board that advertises "ECC memory suitable for mission critical tasks" as its second bullet point, so, I really hope they're also not just running them as non-ECC: https://www.asus.com/us/motherboards-components/motherboards/workstation/pro-ws-x570-ace/ Eletriarnation fucked around with this message at 15:50 on Aug 17, 2023 |
|
#
?
Aug 17, 2023 15:37
|
|
|
shame on an IGA posted:oof somebody needs to build https://shucks.top for canada At a glance looking at the other link it seems like the external storage drive seems to be a good price? Its about 5$ more expensive per TB but is also affordable at around 120$ while the cheaper per TB option is like 350$ which I'm not spending right now and is overkill. 6-8TB is probably basically enough to cover my needs for the forseeable future and at that price I can get two.
|
|
#
?
Aug 17, 2023 16:48
|
|
|
Raenir Salazar posted:At a glance looking at the other link it seems like the external storage drive seems to be a good price? Its about 5$ more expensive per TB but is also affordable at around 120$ while the cheaper per TB option is like 350$ which I'm not spending right now and is overkill. 6-8TB is probably basically enough to cover my needs for the forseeable future and at that price I can get two. That model seems to use SMR drive instead of CMR. It will probably work as a backup drive but wouldn't be good with you future NAS. I'd try to search for an external with shuckable CMR drive so I could reuse it with the NAS.
|
|
#
?
Aug 17, 2023 17:40
|
|
|
Saukkis posted:That model seems to use SMR drive instead of CMR. It will probably work as a backup drive but wouldn't be good with you future NAS. I'd try to search for an external with shuckable CMR drive so I could reuse it with the NAS. Alright so I'm fine if I don't care too much about reusing it later it is likely fine? If I want to see whether a drive is SMR vs CMR is there a way of seeing this information? I looked at both the affordable external WD drives and did ctrl+f for SMR and CMR but didn't see anything.
|
|
#
?
Aug 17, 2023 18:03
|
|
|
It'll do the job, just forget about using it in a NAS.
|
|
#
?
Aug 17, 2023 18:06
|
|
|
Raenir Salazar posted:Alright so I'm fine if I don't care too much about reusing it later it is likely fine? If I want to see whether a drive is SMR vs CMR is there a way of seeing this information? I looked at both the affordable external WD drives and did ctrl+f for SMR and CMR but didn't see anything. This is one source.
|
|
#
?
Aug 17, 2023 18:33
|
|
|
|
| # ? May 28, 2024 23:29 |
|
|
I'm just dicking around with the TrueNAS Cobia beta in a VM. 1) For one, k3s isn't hogging a whole CPU core anymore, now it's just using 2-3% of a VM core. I guess I'll relent and ease up on this k3s stuff, because... 2) I guess the TrueNAS folks learned from the TrueCharts bullshit. Because I can tell it to install a Custom App and just feed it a Docker image name and it goes to install it without bitching. No TrueCharts Docker-Compose app needed.
|
|
#
?
Aug 17, 2023 20:29
|
|