NAS/Storage Megathread: What is this "File Deletion" You Speak of

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›845 »

SolusLunes: Oct 10, 2011; I now have several regrets.

VostokProgram posted:

nit: Pandora is the one who opens the box, she isn't in it

yet

# ? Aug 19, 2022 19:44

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 17:13

Coredump: Dec 1, 2002

Maneki Neko posted:

I ran a storage spaces setup for about 6 years and never had an issue. I eventually moved away from it in the last year or so primarily because the whole hardware setup was getting old and I just moved to unraid instead on a new system.

Ask me about a Server 2019 update that made my storage pool and external drive (both formatted in ReFS) show up as raw. Not a happy day on that one.

# ? Aug 21, 2022 06:56

Eletriarnation: Apr 6, 2005; People don't appreciate the substance of things...
objects in space.; Oven Wrangler

Eletriarnation posted:

I haven't ever thought about thumbnails before and my library is 20TB, so after reading this last night I checked and apparently with them disabled my Plex directory was around 10GB. I enabled thumbnails and today my Plex directory is 20GB - I assume it's done already, although I don't really see any way to tell the status. It shouldn't really matter unless it gets a lot larger though since the system drive has 350GB of free space.

Following up on this from a week ago, just in case anyone is curious. The thumbnails for my actually around 17TB/17,500 files of Plex library seem to have finished generating at around 175GB. Very nice coincidence on the digits, really. It's a lot of data but Plex is 90% of what that PC does and it has a 500GB NVMe system drive, so I might as well let it stay.

# ? Aug 22, 2022 22:21

AlternateAccount: Apr 25, 2005; FYGM

Using Unraid, is there a way to see what processes are sending traffic across the network? I have this weird low baseline of traffic and just wanna know what it is.

ALSO, high shfs CPU usage? Seems to always been tagging the CPU very hard, even when nothing is really going on.

# ? Aug 23, 2022 18:31

BlankSystemDaemon: Mar 13, 2009

AlternateAccount posted:

Using Unraid, is there a way to see what processes are sending traffic across the network? I have this weird low baseline of traffic and just wanna know what it is.

ALSO, high shfs CPU usage? Seems to always been tagging the CPU very hard, even when nothing is really going on.

I don't use unraid, but the latter - assuming it's sshfs - sounds like aesni isn't being used?

# ? Aug 23, 2022 19:41

AlternateAccount: Apr 25, 2005; FYGM

BlankSystemDaemon posted:

I don't use unraid, but the latter - assuming it's sshfs - sounds like aesni isn't being used?

Interesting. Internet suggest this. Seems ok?

code:

root@unraid:~# grep -m1 -o aes /proc/cpuinfo
aes

# ? Aug 23, 2022 19:46

CopperHound: Feb 14, 2012

BlankSystemDaemon posted:

I don't use unraid, but the latter - assuming it's sshfs - sounds like aesni isn't being used?

Shfs not sshfs

shfs is unraid's magic glue that makes the user filesystem work.

# ? Aug 23, 2022 19:51

BlankSystemDaemon: Mar 13, 2009

AlternateAccount posted:

Interesting. Internet suggest this. Seems ok?
code:
root@unraid:~# grep -m1 -o aes /proc/cpuinfo
aes

Caveats about not using Linux aside, that at least suggests the CPU has it enabled; sshfs is part of fuse, so that shouldn't be where the problem is either.

Can you check what kind of speeds you get using openssl speed -elapsed -evp?
You can force it off by setting the OPENSSL_ia32cap environment variable to "~0x200000200000000".

CopperHound posted:

Shfs not sshfs

shfs is unraid's magic glue that makes the user filesystem work.

Well that's confusing! :mad:

# ? Aug 23, 2022 19:55

AlternateAccount: Apr 25, 2005; FYGM

BlankSystemDaemon posted:

Well that's confusing!

Got me, too. The array is encrypted, but the Very Hard Working Process is plain shfs. Even a small Time Machine backup pegs all four cores to 60-70%+. It's an i5-4590, so nothing insane, but it doesn't seem like it should be sweating this hard.

# ? Aug 23, 2022 20:01

BlankSystemDaemon: Mar 13, 2009

AlternateAccount posted:

Got me, too. The array is encrypted, but the Very Hard Working Process is plain shfs. Even a small Time Machine backup pegs all four cores to 60-70%+. It's an i5-4590, so nothing insane, but it doesn't seem like it should be sweating this hard.

Any filesystem with 60-70% of cputime is doing a LOT of things wrong.

# ? Aug 23, 2022 20:14

AlternateAccount: Apr 25, 2005; FYGM

I see posts on their forums about it, but there doesn't seem to be any kind of established cause/fix. Someone mentions IOWAIT being a problem, random grab from top looks like:

code:

%Cpu(s):  3.2 us,  4.9 sy,  0.0 ni, 44.7 id, 46.8 wa,  0.0 hi,  0.4 si,  0.0 st

That does seem quite high. Maybe I have a failing disk that SMART/parity aren't seeing?

# ? Aug 23, 2022 20:31

AlternateAccount: Apr 25, 2005; FYGM

I thiiiink this is related to using Samsung cache disks formatted BTRFS. Removing the cache disk outright seemed to make things way better, IOWAIT never got above 5 instead of 50.
Bad news is that it refuses to add the SSD as a cache without formatting as BTRFS. So I might just do without until I get something else to try.

edit: also might be related to using more Docker containers and those were living on the cache disk and making it do more work.

edit edit: maybe. IOWAIT is down, shfs is still eating between 25-75% of CPU because my Mac wants to run an hourly TM backup. What poo poo :\

AlternateAccount fucked around with this message at 23:08 on Aug 23, 2022

# ? Aug 23, 2022 21:14

IOwnCalculus: Apr 2, 2003

Just dropping this here in case anyone else ever makes the same purchase I did and runs into the same issue.

I bought some used HGST 10TB SAS drives on eBay, which were reclaimed out of some form of Sun storage appliance given the "H7210A520SUN010T" part number it identifies as to the OS. They're also labeled on the drive (and often sold as) HGST HUH72010AL5200, and I suspect the only real difference is the default block size being 528 bytes on the Sun part number instead of either 512 or 4096.

That isn't itself a giant problem, unless you want to treat it as any other 10TB drive, because as delivered it reports the capacity as 9.79TB instead of 10TB.

Someone on serverfault claimed to have success with just running:

code:

sg_format --format --size=512 --fmtpinfo=0 /dev/yourdisk

However, every time I did that (at 12+ hours per run) it would still come back as 9.79TB. I made two changes at once so I don't know which of these changes actually forced the resize, but changing the blocksize to 4096 and adding "count=-1" worked:

code:

sg_format --format --size=4096 --count=-1 --fmtpinfo=0 /dev/yourdisk

code:

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              H7210A520SUN010T
Revision:             A680
Compliance:           SPC-4
User Capacity:        10,000,831,348,736 bytes [10.0 TB]
Logical block size:   4096 bytes

# ? Aug 24, 2022 17:42

e.pilot: Nov 20, 2011; sometimes maybe good
sometimes maybe shit

AlternateAccount posted:

Using Unraid, is there a way to see what processes are sending traffic across the network? I have this weird low baseline of traffic and just wanna know what it is.

ALSO, high shfs CPU usage? Seems to always been tagging the CPU very hard, even when nothing is really going on.

are you using docker containers? could give each container a discrete IP and work backwards from there

# ? Aug 25, 2022 06:26

Olympic Mathlete: Feb 25, 2011

NAS boffins, I currently have a single 4TB Seagate IronWolf in my Synology and I'm going to slam a second drive in to get some redundancy going. I know up thread someone mentioned if you're getting the same HDD to get one from somewhere else so there's a better chance of them not being from the same potentially defective run. I assume I can just slam another brand of drive in there anyway, but what are the pros/cons of doing so?

# ? Aug 25, 2022 15:22

BlankSystemDaemon: Mar 13, 2009

Olympic Mathlete posted:

NAS boffins, I currently have a single 4TB Seagate IronWolf in my Synology and I'm going to slam a second drive in to get some redundancy going. I know up thread someone mentioned if you're getting the same HDD to get one from somewhere else so there's a better chance of them not being from the same potentially defective run. I assume I can just slam another brand of drive in there anyway, but what are the pros/cons of doing so?

Going with a mix of vendors (although there are less of them than there used to be) is much better than attempting to get drives from the same vendor but different production lines (which is the idea behind getting disks that have serial numbers which aren't close to each other).

I don't really know of any cons, if you're buying disk made for the same market from different vendors that won't apply to just buying disks in general.

BlankSystemDaemon fucked around with this message at 16:09 on Aug 25, 2022

# ? Aug 25, 2022 16:07

Canned Sunshine: Nov 20, 2005; CAUTION: POST QUALITY UNDER CONSTRUCTION

Sorry for the dumb question, but I've been trying to do research, and just wanted thoughts/opinions.

My home is currently a mix of Macs and PCs, and in addition to backups to the cloud, I've been using a physical drive as well as the drive in my Apple Time Capsule. The Time Capsule at this point is used as an AP extender to my mesh network.

The Time Capsule is almost out of space, and so I've begun looking at larger storage options. I actually have 2x 8 TB WD Red HDDs that I'd like to use for either NAS or a DAS that would be attached to the Time Capsule or one of the mesh routers. Th idea of moving over to a NAS is pretty appealing though at the moment.

For something that would mostly just be desktop backups and basic file transfers, would something like the Synology DS220j be sufficient, or should I be looking more at DS220+ or something even greater?

Thanks!

# ? Aug 29, 2022 17:31

Bobstar: Feb 8, 2006; KartooshFace, you are not responding efficiently!

SourKraut posted:

Sorry for the dumb question, but I've been trying to do research, and just wanted thoughts/opinions.

My home is currently a mix of Macs and PCs, and in addition to backups to the cloud, I've been using a physical drive as well as the drive in my Apple Time Capsule. The Time Capsule at this point is used as an AP extender to my mesh network.

The Time Capsule is almost out of space, and so I've begun looking at larger storage options. I actually have 2x 8 TB WD Red HDDs that I'd like to use for either NAS or a DAS that would be attached to the Time Capsule or one of the mesh routers. Th idea of moving over to a NAS is pretty appealing though at the moment.

For something that would mostly just be desktop backups and basic file transfers, would something like the Synology DS220j be sufficient, or should I be looking more at DS220+ or something even greater?

Thanks!

I'm regretting getting a -j because it can't run Docker containers. Should have future-proofed with a beefier one. But for just backups and stuff it's fine.

I have a setup like you describe, with laptops backing up to the NAS via Synology Drive. It's not quite as "just works" on the Mac end as Time Machine, but it works fine, I've got hourly backups with versioning. And then the entire NAS backs up to the cloud nightly.

# ? Aug 29, 2022 20:08

Smashing Link: Jul 8, 2003; I'll keep chucking bombs at you til you fall off that ledge!; Grimey Drawer

SourKraut posted:

Sorry for the dumb question, but I've been trying to do research, and just wanted thoughts/opinions.

My home is currently a mix of Macs and PCs, and in addition to backups to the cloud, I've been using a physical drive as well as the drive in my Apple Time Capsule. The Time Capsule at this point is used as an AP extender to my mesh network.

The Time Capsule is almost out of space, and so I've begun looking at larger storage options. I actually have 2x 8 TB WD Red HDDs that I'd like to use for either NAS or a DAS that would be attached to the Time Capsule or one of the mesh routers. Th idea of moving over to a NAS is pretty appealing though at the moment.

For something that would mostly just be desktop backups and basic file transfers, would something like the Synology DS220j be sufficient, or should I be looking more at DS220+ or something even greater?

Thanks!

The Synology j series is ideal for what you are describing. That was my first NAS after I switched from Time Capsules about 8-9 years ago. Impressed your unit is still holding up.

# ? Aug 29, 2022 20:11

Flipperwaldt: Nov 11, 2011; Won't somebody think of the starving hamsters in China?

If you go through my posts in this thread, you can see me try to convince myself that one after the other of (ancient) j series were adequate, find out that you can only somewhat counterbalance the meagre specs by filling it up with an ssd, then renouncing all that when I got a 220+. It all more or less functioned, but now it simply works. Wouldn't characterize it as cheap, but I'm happier.

# ? Aug 29, 2022 20:28

Thanks Ants: May 21, 2004; #essereFerrari

Synology go really tight on the specs when you look at what the boxes cost, I really wouldn't want to use a J-series box for anything other than somewhere your backups synced.

# ? Aug 29, 2022 20:54

Canned Sunshine: Nov 20, 2005; CAUTION: POST QUALITY UNDER CONSTRUCTION

Thank you all, I'll go with the DS220+ then to give it a little more smoothness!

Smashing Link posted:

The Synology j series is ideal for what you are describing. That was my first NAS after I switched from Time Capsules about 8-9 years ago. Impressed your unit is still holding up.

The TC has pretty much just lived its life on my home office desk in a well air conditioned area, but yeah, its longevity is part of why I want a new, larger, more substantial option.

# ? Aug 29, 2022 22:37

AlternateAccount: Apr 25, 2005; FYGM

e.pilot posted:

are you using docker containers? could give each container a discrete IP and work backwards from there

I just dumpstered the whole thing and rebuilt with a different 8700K board/cpu/ram. Seems muuuuuuuch happier.

# ? Aug 29, 2022 23:02

Flipperwaldt: Nov 11, 2011; Won't somebody think of the starving hamsters in China?

SourKraut posted:

Thank you all, I'll go with the DS220+ then to give it a little more smoothness!

If you're ever in a position to need it (not saying you will), you can get cheap off brand ram upgrades for it. Any one of the sellers will mark these specific modules "D4NESO". Read reviews though. Just letting you know those exist, because the markup on the Synology branded ones is kinda nuts.

# ? Aug 29, 2022 23:14

Mr. Crow: May 22, 2008; Snap City mayor for life

I have a usb hard drive enclosure that I use to easily copy data from disks etc. I, an idiot, jammed in a HDD backwards and bent the little sata/power connector in the enclosure. I am not sure if I actually broke it or not and this thing has given me different problems in the past and I don't use it often enough to remember all the issues; but I was able to push the little connection piece down a bit and was able to slide the HDD in and the OS can at least see the block device (/dev/sde).

My question and what's weird is that I can't get the OS to see the partition on it so I can pull data off of the drive. Fdisk seems to be able to read the partition table fine?

code:

# fdisk -l /dev/sde
Disk /dev/sde: 3.64 TiB, 4000787030016 bytes, 976754646 sectors
Disk model:                 
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 33550336 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start        End    Sectors Size Id Type
/dev/sde1           1 4294967295 4294967295  16T ee GPT

partprobe succeeds?

code:

# partprobe -s /dev/sde
/dev/sde: msdos partitions

But the OS still sees nothing (no dmesg output and nothing in lsblk)? I've also tried partx -a /dev/sde though that also does nothing... Any ideas? Is my enclosure just hosed and giving bad data... sometimes?

The only thing I'm curious about is it is an XFS partition and I'm not sure the drive was ever fixed for the 2038 problem? The server is a newer Fedora kernel so maybe there is some issue there? Grasping at straws as I wanted to kick this data copy off and I don't really have spare cables or anything, will probably need to buy a new enclosure and wait for it to arrive...

edit: actually I just noticed the size is way off on the fdisk output, so maybe hosed...

edit2: Well I plugged a different hard drive into the enclosure and it popped up just fine... I had just run a bunch of SMART checks etc. on this disk before popping it out of the server and everything checked out so I don't think the disk is bad?

dmesg output from the "bad" drive, for posterity

code:

kernel: usb 6-3.1: USB disconnect, device number 11
kernel: usb 6-3.1: new SuperSpeed USB device number 12 using xhci_hcd
kernel: usb 6-3.1: New USB device found, idVendor=152d, idProduct=1561, bcdDevice= 1.04
kernel: usb 6-3.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
kernel: usb 6-3.1: Product: SABRENT
kernel: usb 6-3.1: Manufacturer: SABRENT
kernel: usb 6-3.1: SerialNumber: DB98765432143
kernel: scsi host12: uas
kernel: scsi 12:0:0:0: Direct-Access     SABRENT                   0104 PQ: 0 ANSI: 6    
kernel: sd 12:0:0:0: Attached scsi generic sg5 type 0
kernel: sd 12:0:0:0: [sdf] Spinning up disk...
kernel: .ready
kernel: sd 12:0:0:0: [sdf] 976754646 4096-byte logical blocks: (4.00 TB/3.64 TiB)
kernel: sd 12:0:0:0: [sdf] Write Protect is off
kernel: sd 12:0:0:0: [sdf] Mode Sense: 53 00 00 08
kernel: sd 12:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
kernel: sd 12:0:0:0: [sdf] Optimal transfer size 33550336 bytes
kernel: sd 12:0:0:0: [sdf] Attached SCSI disk

Mr. Crow fucked around with this message at 04:42 on Aug 30, 2022

# ? Aug 30, 2022 03:58

Paul MaudDib: May 3, 2006; TEAM NVIDIA:
FORUM POLICE

ok, which versions of mellanox connect-x have good support these days? Obviously *nix support is immortal and they work fine on linux/bsd, but weren't there some cliffs with older mellanox gens where they weren't supported on 10+ and were missing some features even if you installed older versions etc? was it connectx2, bad, connectx3 and up, good? or has connectx3 gotten the boot nowadays too?

# ? Sep 2, 2022 05:08

Wibla: Feb 16, 2011

I have a very early connect-x 10g card in my win10 pc and it works fine.

# ? Sep 2, 2022 08:13

AlternateAccount: Apr 25, 2005; FYGM

I assume 198 Offline Uncorrectable sectors creeping up slowly on a disk means it needs to go in the garbage, right? No other resolution available?

# ? Sep 2, 2022 18:18

BlankSystemDaemon: Mar 13, 2009

AlternateAccount posted:

I assume 198 Offline Uncorrectable sectors creeping up slowly on a disk means it needs to go in the garbage, right? No other resolution available?

It means the disk shortly will or already has run out of spare sectors to move data to, so if you haven't already encountered data corruption you will soon.
Migrate the data off the disk as soon as possible.

# ? Sep 2, 2022 18:21

Combat Pretzel: Jun 23, 2004; No, seriously... what kurds?!

Paul MaudDib posted:

or has connectx3 gotten the boot nowadays too?

I have the ConnectX-3. Works fine. Windows comes with a built-in driver for it, signed by Microsoft themselves. It suspiciously follows Mellanox/NVidia's versioning scheme of WinOF, make of that what you will.

I've been looking for used ConnectX-4's, because the RDMA packet size is doubled, hoping for some more performance and feature proofing. But they're still pretty expensive. And PCIe 3.0.

# ? Sep 3, 2022 00:00

other people: Jun 27, 2004; Associate Christ

Hi I just ordered an Intel NUC 11 Enthusiast system and I want to attach two 3.5" disks that will be used in software raid 1 (mirroring). I already have the disks. Can someone recommend a simple enclosure for this?

My local seller has this ICY BOX thing for �91: https://icybox.de/en/product.php?id=176. It supports USB 3.1 Gen2 so that's good...

It would be cool to have thunderbolt support but those enclosures are at least twice as much and I am not really trying to break any speed records so meh.

# ? Sep 5, 2022 11:18

e.pilot: Nov 20, 2011; sometimes maybe good
sometimes maybe shit

If it�s spinning disks thunderbolt wouldn�t give you any speed advantages over USB.

# ? Sep 5, 2022 11:35

Thanks Ants: May 21, 2004; #essereFerrari

I was about to ask a similar question, I am looking to get something a bit less power hungry than my existing (old) Microserver and was considering a NUC with a USB enclosure. Lots of bad reviews of generic AliExpress-grade boxes so hopefully the Icy Box presenting two individual drives will do the trick.

# ? Sep 5, 2022 12:34

other people: Jun 27, 2004; Associate Christ

e.pilot posted:

If it�s spinning disks thunderbolt wouldn�t give you any speed advantages over USB.

Oh I guess that was obvious :doh:

.

There is another ICY BOX (https://icybox.de/en/product.php?id=445) that is basically the same features but the shell is aluminum instead of plastic. It sells for about �50 more around here. Neither offer any control of the 60mm fan speed and at least a few reviews say it is noisy.

For about the same price as the aluminum icy box is this Raidon thing: https://www.raidon.com.tw/RAIDON2016/product.php?id=182 but I have never heard of them and can't find any reviews. It has two fan speeds though!

# ? Sep 5, 2022 13:47

BlankSystemDaemon: Mar 13, 2009

Ihmemies posted:

I want to get rid of Spotify. It is really getting to my nerves with all the crapware and hostile features. What are my options? Like, I want to host my own private Spotify for myself. I can probably make a PC with VM's and run some OS on one VM.

Which OS would be good for:
1. host audio files
2. stream said files to different devices

Which software would be good to listen to the music in different devices (android, windows etc).?

There is Plex for video content. Is there anything similar for audio content? Thanks.

I haven't' used it myself so I can't speak to how good it is, but Plex also does audio (PlexAmp).

# ? Sep 7, 2022 17:21

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 17:13

Sir Bobert Fishbone: Jan 16, 2006; Beebort

Scruff McGruff posted:

I haven't' used it myself so I can't speak to how good it is, but Plex also does audio (PlexAmp).

Plex & PlexAmp has completely replaced Spotify for me.

# ? Sep 7, 2022 17:36

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›845 »