NAS/Storage Megathread: What is this "File Deletion" You Speak of

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›847 »

hogofwar: Jun 25, 2011; 'We've strayed into a zone with a high magical index,' he said. 'Don't ask me how. Once upon a time a really powerful magic field must have been generated here, and we're feeling the after-effects.'
'Precisely,' said a passing bush.

I think the question will be, do you intend to have Plex transcoding?

If you need transcoding the general advice is to get a Intel processor from the last few generations with quicksync. Even an i3 would probably be decent enough.

If you just want to do direct play, anything will work.

# ? Aug 26, 2023 16:51

Adbot: ADBOT LOVES YOU

# ? May 28, 2024 10:30

BlankSystemDaemon: Mar 13, 2009

A new option for transcoding is the Intel Arc A380 from AsRock, which gets you a dual-slot low-profile GPU with full AV1 encoding and decoding, for as low as $99.

# ? Aug 26, 2023 17:05

Corb3t: Jun 7, 2003

BlankSystemDaemon posted:

A new option for transcoding is the Intel Arc A380 from AsRock, which gets you a dual-slot low-profile GPU with full AV1 encoding and decoding, for as low as $99.

How are the Linux and Plex drivers on these Intel GPUs nowadays? What about AV1 support? That's super tempting for the price point.

# ? Aug 26, 2023 17:27

Saukkis: May 16, 2003; Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.

surfacelevelspeck posted:

I'm planning to use Ubuntu 20.04 to host Plex so hopefully I can just keep that on an external drive, or if it's a better choice an m.2 on the motherboard.

Why are you planning to use a distro with such a limited lifespan?

# ? Aug 26, 2023 19:01

A Bag of Milk: Jul 3, 2007; I don't see any American dream; I see an American nightmare.

I installed a Debian 12 VM on TrueNAS Scale 22.12.2.3. Could somebody let me know how to paste from clipboard? I dunno if the internet has degraded or its just me, but I cannot figure this out and nothing seems to be pointing me in the right direction. Looks like this:

# ? Aug 26, 2023 19:33

BlankSystemDaemon: Mar 13, 2009

Corb3t posted:

How are the Linux and Plex drivers on these Intel GPUs nowadays? What about AV1 support? That's super tempting for the price point.

You're asking the wrong person about that; try the Linux thread or the HTPC thread.

# ? Aug 26, 2023 20:54

surfacelevelspeck: Oct 1, 2008; communism's sleepiest soldier

Saukkis posted:

Why are you planning to use a distro with such a limited lifespan?

I mean I was only planning it because it was what I found looking around for easy Plex setups, if there's a better (free) choice I'd be more than happy to do that instead.

# ? Aug 26, 2023 20:58

Corb3t: Jun 7, 2003

BlankSystemDaemon posted:

You're asking the wrong person about that; try the Linux thread or the HTPC thread.

Or the Plex thread, but I think it's fair to ask a user who recommends a specific GPU if it actually does a good job transcoding within the confines of the user's needs (Linux + Plex).

# ? Aug 26, 2023 21:12

BlankSystemDaemon: Mar 13, 2009

Corb3t posted:

Or the Plex thread, but I think it's fair to ask a user who recommends a specific GPU if it actually does a good job transcoding within the confines of the user's needs (Linux + Plex).

Yes, but I don't use Linux or Plex - I use FreeBSD and Kodi.

FreeBSD does get the video drivers for Intel and AMD cards from Linux, and while they work well the recommendation is more of a general heads-up since I can't buy one myself (it costs the equivalent of ~$160, which I don't have).
Plex also notoriously doesn't really play well with all forms of transcoding, at least historically - whereas Kodi has no issues just supporting whatever libva and libavcodec implements.

# ? Aug 26, 2023 21:24

Saukkis: May 16, 2003; Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.

surfacelevelspeck posted:

I mean I was only planning it because it was what I found looking around for easy Plex setups, if there's a better (free) choice I'd be more than happy to do that instead.

I assume it will work identically on Ubuntu 22.04.

https://linuxhint.com/install_plex_ubuntu-2/

# ? Aug 26, 2023 22:47

Nitrousoxide: May 30, 2011; do not buy a oneplus phone

You would probably have a much easier time just using podman or docker.

# ? Aug 26, 2023 22:53

Aware: Nov 18, 2003

Corb3t posted:

How are the Linux and Plex drivers on these Intel GPUs nowadays? What about AV1 support? That's super tempting for the price point.

Anecdotally from Reddit it has good support in Linux and Plex.

# ? Aug 26, 2023 23:28

Smashing Link: Jul 8, 2003; I'll keep chucking bombs at you til you fall off that ledge!; Grimey Drawer

Well I finally got around to spinning up a TrueNAS Core VM on my Unraid box to try out ZFS replication (other TrueNAS server -> TrueNAS VM) and it's humming along smoothly. Quite an easy setup. I think this is one area where TrueNAS has a definite advantage over Unraid.

# ? Aug 27, 2023 20:24

hogofwar: Jun 25, 2011; 'We've strayed into a zone with a high magical index,' he said. 'Don't ask me how. Once upon a time a really powerful magic field must have been generated here, and we're feeling the after-effects.'
'Precisely,' said a passing bush.

The only thing keeping me away from TrueNAS (and zfs in general) is the general inability to expand the storage, unless you double the storage I believe? I saw there has been work on expansion for years now and it still looks a bit far off still. One day

# ? Aug 27, 2023 20:46

Wibla: Feb 16, 2011

Funny you should mention that :v:

That said - I am probably going to ditch TrueNAS Scale for proxmox.

# ? Aug 27, 2023 20:56

Smashing Link: Jul 8, 2003; I'll keep chucking bombs at you til you fall off that ledge!; Grimey Drawer

Wibla posted:

Funny you should mention that

That said - I am probably going to ditch TrueNAS Scale for proxmox.

I have been hearing about that but not sure if you will be able to mix and match disk sizes.

I may go ahead and spin up a proxmox vm to play around with but I imagine if I keep trying to pass through hardware to successively nested VMs something will eventually poo poo the bed. But who knows.

# ? Aug 27, 2023 21:07

Combat Pretzel: Jun 23, 2004; No, seriously... what kurds?!

The new disk will need to be same or bigger than the existing bunch, but the smallest disk in the array will define how much disk space from the new disk will be available.

# ? Aug 27, 2023 21:19

IOwnCalculus: Apr 2, 2003

You can mix and match disk sizes today, but the disk will be treated the same as the smallest disk in the same vdev.

Once you replace all disks in a vdev with larger ones, you can expand it to the new size.

e: f,b

# ? Aug 27, 2023 21:19

Wibla: Feb 16, 2011

I don't think you'll be able to mix and match freely, no. This mainly seems to be to expand a vdev with a new drive of same or bigger capacity.

e:f,b :haw:

# ? Aug 27, 2023 21:20

BlankSystemDaemon: Mar 13, 2009

All of the implementations for doing the kind of arbitrary disk mixing that people want are, as far as I know, proprietary (meaning there's no way to know if they work like they claim to work, except to trust your data to them and find out if you were wrong the hard way) and they're also not copy-on-write or transactionally atomic with checksumming (meaning they offer little in the way of data integrity).

I'm not sure I can envision a way to make ZFS operate the way people want it to work, without first implementing block pointer rewrite - which, while it's a feature that's often mentioned, probably won't ever actually land on any kind of reasonable time-frame.
Ahrens has basically said that in order to implement BPR, every single other feature would have to be implemented first - because adding BPR will increase the complexity of the code so much, that any future maintenance of the code will be so difficult as to make it not worth implementing.

# ? Aug 27, 2023 22:15

hogofwar: Jun 25, 2011; 'We've strayed into a zone with a high magical index,' he said. 'Don't ask me how. Once upon a time a really powerful magic field must have been generated here, and we're feeling the after-effects.'
'Precisely,' said a passing bush.

Wibla posted:

Funny you should mention that

That said - I am probably going to ditch TrueNAS Scale for proxmox.

Yep, saw that before. The previous PR for it has been going on for years and my general understanding is that this is still a long while off. One day

# ? Aug 27, 2023 23:14

Yaoi Gagarin: Feb 20, 2014

Btrfs can do arbitrary disk mixing, but it has the write hole problem :whitewater:

# ? Aug 27, 2023 23:35

BlankSystemDaemon: Mar 13, 2009

hogofwar posted:

Yep, saw that before. The previous PR for it has been going on for years and my general understanding is that this is still a long while off. One day

Don took it over recently with the intent of reviewing the entire implementation and getting it landed, after it got stuck - and he seems pretty focused on it.

I can't make any promises on his behalf, of course - but I think the plan to get it into 3.0 is still on?

VostokProgram posted:

Btrfs can do arbitrary disk mixing, but it has the write hole problem

BTRFS also has the slightly bigger issue, in that its primary developers solution to any sort of problem with it is to take the system out back and put a bullet through its brain: Facebook are exclusively using it for their scale-out load-balancing system where the practice is to simply shut down hosts if they experience any issues, as persistent storage is handled by Tectonic, a proprietary filesystem they've developed in-house.

# ? Aug 28, 2023 00:03

Yaoi Gagarin: Feb 20, 2014

maybe bcachefs will be the one

# ? Aug 28, 2023 00:06

BlankSystemDaemon: Mar 13, 2009

VostokProgram posted:

maybe bcachefs will be the one

Aaany day now, right?

# ? Aug 28, 2023 00:12

hogofwar: Jun 25, 2011; 'We've strayed into a zone with a high magical index,' he said. 'Don't ask me how. Once upon a time a really powerful magic field must have been generated here, and we're feeling the after-effects.'
'Precisely,' said a passing bush.

If you just want to use the nas functionality of TrueNAS, is there any reason to pick scale over core or vice versa?

# ? Aug 29, 2023 00:25

power crystals: Jun 6, 2007; Who wants a belly rub??

Core is more mature, Scale is where their attention seems to be focused. Scale is more easily added to later if you think you might some day want more than just a NAS, but for "a bunch of files on a server" there's not a tremendously compelling reason to pick either, at least that I've seen.

# ? Aug 29, 2023 01:06

Eletriarnation: Apr 6, 2005; People don't appreciate the substance of things...
objects in space.; Oven Wrangler

Depending on how esoteric your hardware is, one might have better driver support than the other. The official docs state that Ryzen platforms are better supported on Linux than FreeBSD and therefore should prefer Scale, but folks in this thread have said they run Core on Ryzen without any issues and it's not clear if there's any specific problem which would lead the docs to this conclusion.

I'm not aware of any specific cases that are known to be a problem, but if you have a weird HBA/NIC/whatever it could be worth checking on.

Eletriarnation fucked around with this message at 01:44 on Aug 29, 2023

# ? Aug 29, 2023 01:34

BlankSystemDaemon: Mar 13, 2009

Eletriarnation posted:

Depending on how esoteric your hardware is, one might have better driver support than the other. The official docs state that Ryzen platforms are better supported on Linux than FreeBSD and therefore should prefer Scale, but folks in this thread have said they run Core on Ryzen without any issues and it's not clear if there's any specific problem which would lead the docs to this conclusion.

I'm not aware of any specific cases that are known to be a problem, but if you have a weird HBA/NIC/whatever it could be worth checking on.

I think Linux has more attempts at fixing the Zen 1 errata (of which there is quite a bit, including some very serious ones), but since there's now Zenbleed which requires mitigations that'll probably affect performance by 50% or more, I think most people would want to avoid Zen through Zen 2.

# ? Aug 29, 2023 06:47

Ihmemies: Oct 6, 2012

I'm trying to do a Backblaze B2 Cloud Storage Bucket backup for my photos. They are around 1TB which amounts to maybe $5/month for cloud backup. I added a Backblaze API key to my truenas for the photo backup bucket, and added a cloud sync task.

Even a dry run fails:

code:

<5>NOTICE: 
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Elapsed time:         1.6s

<5>NOTICE: 
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Elapsed time:         2.6s

<5>NOTICE: 
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
... 38 more lines ...
<3>ERROR : B2 bucket [bucketname]: error reading destination root directory: Unknown 401  (401 unauthorized)
<3>ERROR : B2 bucket [bucketname]: not deleting files as there were IO errors
<3>ERROR : B2 bucket [bucketname]: not deleting directories as there were IO errors
<3>ERROR : Attempt 3/3 failed with 1 errors and: Unknown 401  (401 unauthorized)
<5>NOTICE: 
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:        11.6s

Failed to sync: Unknown 401  (401 unauthorized)

What does it want?

I get the same error as when trying to edit the folder in cloud sync task:

code:

[EFAULT] <3>ERROR : : error listing: Unknown 401 (401 unauthorized) Failed to lsjson with 2 errors: last error was: error in ListJSON: Unknown 401 (401 unauthorized)

The photo API key should have enough permissions:

code:

capabilities: deleteFiles, listBuckets, writeBucketEncryption, writeBucketReplications, writeFiles

# ? Sep 3, 2023 09:08

Ihmemies: Oct 6, 2012

Can not figure this out. Maybe I should try Azure blob, cold is only $0,0036. Archive even cheaper, but dunno where they store that data, on tape?

# ? Sep 3, 2023 09:33

Harik: Sep 9, 2001; From the hard streets of Moscow
First dog to touch the stars; Plaster Town Cop

VostokProgram posted:

Btrfs can do arbitrary disk mixing, but it has the write hole problem

do not use btrfs raid for any reason. there is no btrfs raid mode that won't gently caress up your data sooner ~~or later~~. btrfs on a raid array is fine and works extremely well. That said, I'm using ZFS on my next pool and retiring at least the smaller btrfs array and possibly the larger as well.

speaking of bad ideas: refurb seagate exos for $10/tb? 1 year ebay warranty and reseller claims to give you 5 years (lol)

https://www.ebay.com/itm/155636746868

40% off buys a lot of spares even if the 5 year warranty is bullshit.

BlankSystemDaemon posted:

I think Linux has more attempts at fixing the Zen 1 errata (of which there is quite a bit, including some very serious ones), but since there's now Zenbleed which requires mitigations that'll probably affect performance by 50% or more, I think most people would want to avoid Zen through Zen 2.

Zen2 only. Zen1, 1+ and 3 don't have the issue.

upcloud found no significant performance inpact

quote:

At the time of publication, AMD released a microcode update for the affected processors. Their mitigation is implemented via the MSR register, which turns off a floating point optimization that otherwise would have allowed a move operation. In our testing, applying this mitigation has not had a detrimental impact on overall server performance.

nobody seems to have benchmarked it, which is weird.

Harik fucked around with this message at 12:48 on Sep 3, 2023

# ? Sep 3, 2023 12:03

Harik: Sep 9, 2001; From the hard streets of Moscow
First dog to touch the stars; Plaster Town Cop

Is there a disk-oriented case besides the meshify-2? Because it doesn't actually come with hardware for anything beyond 6 drives and it's at least another $100 to buy the rest of the drive kits for it.

# ? Sep 3, 2023 12:57

Theophany: Jul 22, 2014; SUCCHIAMI IL MIO CAZZO DA DIETRO, RANA RAGAZZO

2022 FIA Formula 1 WDC

Harik posted:

Is there a disk-oriented case besides the meshify-2? Because it doesn't actually come with hardware for anything beyond 6 drives and it's at least another $100 to buy the rest of the drive kits for it.

Node 804 can do 12 drives before you need to buy extra brackets - 2x 4 drive cages in the back, 2 drives on the chassis floor and 2x 2.5 drives in the front panel. With a bit of fuckery and tight packaging I managed to get 12 3.5" drives in there but the airflow was a real problem. I've since migrated to a Define 7XL but you're back at the problem of needing to buy a bunch of brackets to fully use up the available space.

# ? Sep 3, 2023 13:27

Ihmemies: Oct 6, 2012

Harik posted:

Is there a disk-oriented case besides the meshify-2? Because it doesn't actually come with hardware for anything beyond 6 drives and it's at least another $100 to buy the rest of the drive kits for it.

I got an old Define R5 for 40�. It has 8 �hotswap� bays + some more room in 5,25� slots.

# ? Sep 3, 2023 13:31

IOwnCalculus: Apr 2, 2003

Harik posted:

speaking of bad ideas: refurb seagate exos for $10/tb? 1 year ebay warranty and reseller claims to give you 5 years (lol)

https://www.ebay.com/itm/155636746868

40% off buys a lot of spares even if the 5 year warranty is bullshit.

Goharddrive is legit. I bought some drives from them via Newegg, one died years later, they refunded that drive in full after I shipped it back. I would absolutely use the savings to buy a spare or two, though.

I bought a bunch of 10TB drives from them recently but none of those have died yet.

# ? Sep 3, 2023 14:47

TITTIEKISSER69: Mar 19, 2005; SAVE THE BEES
PLANT MORE TREES
CLEAN THE SEAS
KISS TITTIESS

TITTIEKISSER69 posted:

Looking for advice concerning my Synology NAS, which is just a simple DS120j with a single SATA SSD installed.

That drive is just about full, so I got a bigger one. What's the best way to clone old to new?

I have a SATA-USB adapter, can I plug it in there and then use one of the Synology apps to clone it, then shut it down and install the new drive? If so, which app?

Just bumping this up, any advice?

# ? Sep 3, 2023 17:46

Internet Explorer: Jun 1, 2005

I don't have any specific advice for you, but the problem you are going to run into is that the OS on Synology NASes is on the drive(s). You might be able to take the drive out and use a PC to clone. Then put the new drive into the NAS and extend the volume. But the only other option I can think of is essentially starting a new NAS with the new drive and copying your data over. But if you use Synology apps and not just store data, that isn't going to work well.

# ? Sep 3, 2023 18:37

e.pilot: Nov 20, 2011; sometimes maybe good
sometimes maybe shit

Harik posted:

do not use btrfs raid for any reason. there is no btrfs raid mode that won't gently caress up your data sooner ~~or later~~. btrfs on a raid array is fine and works extremely well. That said, I'm using ZFS on my next pool and retiring at least the smaller btrfs array and possibly the larger as well.

speaking of bad ideas: refurb seagate exos for $10/tb? 1 year ebay warranty and reseller claims to give you 5 years (lol)

https://www.ebay.com/itm/155636746868

40% off buys a lot of spares even if the 5 year warranty is bullshit.

Zen2 only. Zen1, 1+ and 3 don't have the issue.

upcloud found no significant performance inpact

nobody seems to have benchmarked it, which is weird.

I run refurb disks in my backup server but they only spin up once a day for my main server backups, replace them as they die.

Been doing this for nearly 4 years now and only one is starting to throw smart errors.

# ? Sep 3, 2023 19:09

Adbot: ADBOT LOVES YOU

# ? May 28, 2024 10:30

Flipperwaldt: Nov 11, 2011; Won't somebody think of the starving hamsters in China?

TITTIEKISSER69 posted:

Just bumping this up, any advice?

There's no established procedure for this as far as Synology is concerned. Cloning can be done on another computer if you can connect both drives at once. I watched this terrible video where a guy goes from a larger to a smaller drive and gets nowhere with a number of softwares for half the video before settling on minitool partition wizard. I think you have to switch the order to cloning, then resizing the partition in partition wizard. I can't even find confirmation you can extend the partition or the volume within the synology os without formatting or indeed that it just happily adopts the new space if you extended the partition outside of the os. So it might or might not even work in that sense.

If the setup is fairly simple and mostly uses first party software, I'd just go through writing down the settings and start over.

# ? Sep 3, 2023 19:18

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›847 »