NAS/Storage Megathread: What is this "File Deletion" You Speak of

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›842 »

YerDa Zabam: Aug 13, 2016

Are they the new fangled dual head ones?

# ? Mar 6, 2024 21:38

Adbot: ADBOT LOVES YOU

# ? Apr 29, 2024 01:49

Combat Pretzel: Jun 23, 2004; No, seriously... what kurds?!

Didn't they act essentially like two drives in one enclosure? At least the SAS ones?

--edit: Wendell has a video about the SATA version. It's basically just concats two disks. Some kind of interleaving would have been interesting to see what it does.

Combat Pretzel fucked around with this message at 22:00 on Mar 6, 2024

# ? Mar 6, 2024 21:57

BlankSystemDaemon: Mar 13, 2009

Truck Stop Daddy posted:

Holy poo poo the exos x18s are loud.

In a datacenter, they're hardly the loudest thing.

# ? Mar 6, 2024 22:10

Twerk from Home: Jan 17, 2009; This avatar brought to you by the 'save our dead gay forums' foundation.

YerDa Zabam posted:

Are they the new fangled dual head ones?

No Exos X18 is just a couple generation old plain old cheap disk.

The dual head ones are branded "2X18" or whatever.

Speaking of HDD tech, I just learned this week that there's an SAS-4, 22.5gbit speeds. Are we getting a SATA-4 in the near future too?

# ? Mar 6, 2024 22:57

Baba Oh Really: May 21, 2005; Get 'ER done

BlankSystemDaemon posted:

In a datacenter, they're hardly the loudest thing.

This. After working in one, you wouldn't give a poo poo about how loud drives are.

# ? Mar 7, 2024 00:01

BlankSystemDaemon: Mar 13, 2009

Twerk from Home posted:

No Exos X18 is just a couple generation old plain old cheap disk.

The dual head ones are branded "2X18" or whatever.

Speaking of HDD tech, I just learned this week that there's an SAS-4, 22.5gbit speeds. Are we getting a SATA-4 in the near future too?

SATAIII is already limited by the single 16byte command queue offered by AHCI, so boosting the bandwidth isn't going to give any massive bandwidth jumps - especially since most consumer SSDs using SATAIII can barely push 550Mbps read/write, which is the limit once you factor 8b/10b line coding.

SAS4 is switching to a packet-based 128b/150b line coding which means you're looking at 20 bits of Reed-Solomon FEC which handles 4 128bit words (and a header) - hence why it's that much faster. They've also switched the PHY so that you can finally get 6m SAS cables - but this also means you'll need brand new cables. It also doesn't really change the number of queues or the length of them, but that's not surprising since spinning rust doesn't really benefit even if you have dual-actuator drives.
Basically, T10 is betting that spinning rust will still be the king of bulk storage for the foreseeable future.

Baba Oh Really posted:

This. After working in one, you wouldn't give a poo poo about how loud drives are.

The threshold for spending any amount of time in a noisy environment is 85dBa, and datacenters are typically measured at 92-95dBa (and remember, +10dBa is a doubling of pressure).
So if you spend any amount of time in a datacenter without hearing protection, you either have or will develop tinnitus.

# ? Mar 7, 2024 00:38

Truck Stop Daddy: Apr 17, 2013; A janitor cleans the bathroom; Muldoon

Yeah, I got 3 normal 18tb x18 drives in a ds423+. Been moving the nas around the house to see if I can find a place where I do not constantly hear it. Haven't been successful so far haha. They were oddly cheaper than any other 18tb drives that were available to me, so all in all I'm happy.

# ? Mar 7, 2024 20:19

Henrik Zetterberg: Dec 7, 2007

Put a 22TB Ironwolf Pro in my DS1821+ to expand my SHR pool and it�s still being processed a week later 🥵

I got 2 more drives to add as well lol

# ? Mar 8, 2024 07:38

Rescue Toaster: Mar 13, 2003

Any suggestions for a backup util that's just slightly more advanced than rsync and easy to setup? (For linux)

I have my NAS and a lot of what's on it I could rebuild/recover/re-rip/whatever, but obviously some would be hard or impossible to replace. I have a couple USB 3.0 drives I want to start swapping periodically. Hopefully eventually I can add a third off-site one.

My ideal plan/setup would be:

* Flag folders on the NAS I want to backup with a file named .backup or something that I could locate with a find command, and I can parse/translate that into whatever format necessary.
* I can set up some automation so when a USB drive is first attached, it kicks off a quick SMART check and then backup process, and at the end dismounts the drive and sends me a push notification. I can setup all this scripting myself easily enough.
* Backup mainly just needs to sync up the drive to whatever in the marked/listed folders is changed.

I could in theory just use rsync even, but I would like some basic compression, and ideally some parity generation similar to par2's or whatever, to help protect against a couple random bad sectors on the backup drive, obviously. Also ideally it would fully read/re-verify/exercise the backup each time it's attached to make sure nothing is getting stale.

I'm curious if anybody in a similar situation (rotating USB drives for backup, NAS too large for backup media so you want to filter it down) has found a solution that works for you.

# ? Mar 9, 2024 03:19

Shumagorath: Jun 6, 2001

BlankSystemDaemon posted:

The threshold for spending any amount of time in a noisy environment is 85dBa, and datacenters are typically measured at 92-95dBa (and remember, +10dBa is a doubling of pressure).
So if you spend any amount of time in a datacenter without hearing protection, you either have or will develop tinnitus.

Thank you for this post; going to buy a few sets of Peltors.

# ? Mar 9, 2024 03:40

Mad Wack: Mar 27, 2008; "The faster you use your cooldowns, the faster you can use them again"

Truck Stop Daddy posted:

Yeah, I got 3 normal 18tb x18 drives in a ds423+. Been moving the nas around the house to see if I can find a place where I do not constantly hear it. Haven't been successful so far haha. They were oddly cheaper than any other 18tb drives that were available to me, so all in all I'm happy.

mine sits in the basement near the water heater and the boiler, they compete to see who can be louder

# ? Mar 9, 2024 05:19

xgalaxy: Jan 27, 2004; i write code

I work from home and use a computer that I can�t upgrade.
It has a 2.5G Ethernet port and USB Super Speed ports on the back marked 10 and 20 - which googling seems to suggest are 10Gbps and 20Gbps capable.

I have the need to have a large amount of data be periodically written and constantly read for work reasons.
Essentially, for work I use a digital art assets management system that can tier data access such that the system will first look for the data on local drive, then a local network share or equivalent, etc, etc, until finally it hits the work VPN to download it to store locally after exhausting faster "caches".

Currently it�s just my work computer with an Nvme drive. But the drive is only 1TB. And because I�m remote hitting the VPN to download isn�t cutting it for me.

I also wouldn�t mind using storage for non-work purposes such as streaming to my living room AppleTv.

What would be good options for my use case and hardware limitations?
Maybe I would be better off buying something like the Samsung T9 for the work computer and doing NAS for the streaming?
The 4TB version might work but smaller than I�d like. Is there something like that but with 10TB?

xgalaxy fucked around with this message at 16:51 on Mar 10, 2024

# ? Mar 10, 2024 04:13

Moey: Oct 22, 2010; I LIKE TO MOVE IT

When it rains, it pours. Another QNAP vuln

https://www.bleepingcomputer.com/news/security/qnap-warns-of-critical-auth-bypass-flaw-in-its-nas-devices/

# ? Mar 10, 2024 16:43

hark: May 10, 2023; I'm sleep

Moey posted:

When it rains, it pours. Another QNAP vuln

https://www.bleepingcomputer.com/news/security/qnap-warns-of-critical-auth-bypass-flaw-in-its-nas-devices/

I feel like I get one of their emails about these like every other week

# ? Mar 11, 2024 03:49

BonoMan: Feb 20, 2002; Jade Ear Joe

Are there actual stats on how much more vulnerable QNAP and Synology devices are over larger "more enterprise" brands?

I feel like I hear that a lot (more about QNAP but def some about Synology too), but it seems like NAS' in general are just going to be constant targets given what the devices actually are and that's naturally going to cause a neverending stream of potential exploits and deployed fixes.

# ? Mar 11, 2024 14:28

Beve Stuscemi: Jun 6, 2001

I havent heard a ton about Synology vulnerabilities? I think its just Qnap?

# ? Mar 11, 2024 14:40

Theophany: Jul 22, 2014; SUCCHIAMI IL MIO CAZZO DA DIETRO, RANA RAGAZZO

2022 FIA Formula 1 WDC

Beve Stuscemi posted:

I havent heard a ton about Synology vulnerabilities? I think its just Qnap?

It may be that Qnap mailshot their customers when the advisories are published, whereas Synology don't and just list them here: https://www.synology.com/en-global/security/advisory

# ? Mar 11, 2024 14:56

Saukkis: May 16, 2003; Unless I'm on the inside curve pointing straight at oncoming traffic the high beams stay on and I laugh at your puny protest flashes.
I am Most Important Man. Most Important Man in the World.

You can use Cvedetails.com to get a clearer picture. It looks like Qnap has had very bad start of the year, or they have done a thorough search for vulnerabilities.

Qnap
Synology

# ? Mar 11, 2024 15:20

Scruff McGruff: Feb 13, 2007; Jesus, kid, you're almost a detective. All you need now is a gun, a gut, and three ex-wives.

It might be a byproduct of the fact that they just launched their bug bounty program last year. It's always tough with these types of announcements, is QNAP really more vulnerable than others or are they just more transparent about when security flaws are found? If they're more vulnerable people should avoid them, but if they're really not and are just more transparent about discovered flaws we should encourage that behavior from companies.

# ? Mar 11, 2024 15:53

History Comes Inside!: Nov 20, 2004

The internet will assume whichever option lets them dunk on the most people for not doing it their preferred way so ultimately it doesn�t really matter.

# ? Mar 11, 2024 15:55

Thanks Ants: May 21, 2004; #essereFerrari

Don't expose your NAS to the internet or enable the feature where it creates a reverse tunnel

# ? Mar 11, 2024 17:46

Wiggly Wayne DDS: Sep 11, 2010

Scruff McGruff posted:

It might be a byproduct of the fact that they just launched their bug bounty program last year. It's always tough with these types of announcements, is QNAP really more vulnerable than others or are they just more transparent about when security flaws are found? If they're more vulnerable people should avoid them, but if they're really not and are just more transparent about discovered flaws we should encourage that behavior from companies.

given the latest patched builds for the worst vuln date back to november/december depending on the os they're just holding off on announcing until everyone's had time to update

code:

We have already fixed the vulnerabilities in the following versions:
Affected Product 	Fixed Version
QTS 5.1.x 		QTS 5.1.3.2578 build 20231110 and later
QTS 4.5.x 		QTS 4.5.4.2627 build 20231225 and later
QuTS hero h5.1.x 	QuTS hero h5.1.3.2578 build 20231110 and later
QuTS hero h4.5.x 	QuTS hero h4.5.4.2626 build 20231225 and later
QuTScloud c5.x 		QuTScloud c5.1.5.2651 and later
myQNAPcloud 1.0.x 	myQNAPcloud 1.0.52 (2023/11/24) and later

# ? Mar 11, 2024 21:57

BlankSystemDaemon: Mar 13, 2009

It's also just sometimes useful to remember all software contains vulnerabilities, even if the developers follow best practices - the only practical difference is whether a company tells you about them or not.

After you account for static analysis, all forms of sanitizers (not all of which are available, especially for proprietary software), fuzzing, and even if you add code review into the mix, there's only so much that can be done when the average programmer introduces one mistake per 100 lines of code (even the best programmers only manage one mistake per 1000 lines of code).
An OS contains tens to a hundred millions of code (Linux is 14 million just for the kernel, a Linux distribution is over 50 million, Windows and macOS is over 100 million by all accounts).

Scruff McGruff posted:

It might be a byproduct of the fact that they just launched their bug bounty program last year. It's always tough with these types of announcements, is QNAP really more vulnerable than others or are they just more transparent about when security flaws are found? If they're more vulnerable people should avoid them, but if they're really not and are just more transparent about discovered flaws we should encourage that behavior from companies.

It's definitely a result of the bug bounty program, yes.

Thanks Ants posted:

Don't expose your NAS to the internet or enable the feature where it creates a reverse tunnel

Don't use a reverse tunnel if it's going somewhere you have no control over, as that just means that once (not if) the PoP you're reverse tunneling to gets taken over by a persistent threat, you're hosed along with everyone else.
This is one of the things that I expect to happen with Plex, which makes me wonder how many people are eventually gonna be hosed once someone decides Plex is a juicy enough target to invest time/money into subverting one of their developers.

Use something that's deliberately designed to be resistant to attacks. Your best option is wireguard or IPsec (both can be done with algo using a passphrase-protected keyfile and/or MFA).

Wiggly Wayne DDS posted:

given the latest patched builds for the worst vuln date back to november/december depending on the os they're just holding off on announcing until everyone's had time to update

code:

We have already fixed the vulnerabilities in the following versions:
Affected Product 	Fixed Version
QTS 5.1.x 		QTS 5.1.3.2578 build 20231110 and later
QTS 4.5.x 		QTS 4.5.4.2627 build 20231225 and later
QuTS hero h5.1.x 	QuTS hero h5.1.3.2578 build 20231110 and later
QuTS hero h4.5.x 	QuTS hero h4.5.4.2626 build 20231225 and later
QuTScloud c5.x 		QuTScloud c5.1.5.2651 and later
myQNAPcloud 1.0.x 	myQNAPcloud 1.0.52 (2023/11/24) and later

This is industry standard practice; everyone should want the fix deployed by the time details are published, unless you're dealing with a zero-day that's already being leveraged in an attack chain.

BlankSystemDaemon fucked around with this message at 22:01 on Mar 11, 2024

# ? Mar 11, 2024 21:58

Henrik Zetterberg: Dec 7, 2007

Suggestions for a UPS that can talk to my 8-disk Synology about gracefully shutting down in the event of a power loss?
And I have a NUC right next to it running Win11. I assume the UPS would have Windows software to do the same? Can you USB 2 machines into 1 UPS with control like that?

I don't need something that has like a 6 hours runtime or whatever. Just long enough for a graceful shutdown.

Henrik Zetterberg fucked around with this message at 22:18 on Mar 13, 2024

# ? Mar 13, 2024 22:13

Thanks Ants: May 21, 2004; #essereFerrari

Pretty much any UPS with a USB port will work as long as you're not buying direct from AliExpress. The Synology can serve the UPS status out over the LAN, you can then connect the Pi to it for graceful shutdown.

It's a Reddit link but https://www.reddit.com/r/synology/comments/gtkjam/use_synology_nas_as_ups_server_to_safely_power/

If you want to be sure the UPS is compatible then the list is at https://www.synology.com/en-au/compatibility?search_by=category&category=upses

# ? Mar 13, 2024 22:42

Harik: Sep 9, 2001; From the hard streets of Moscow
First dog to touch the stars; Plaster Town Cop

xgalaxy posted:

I work from home and use a computer that I can�t upgrade.
It has a 2.5G Ethernet port and USB Super Speed ports on the back marked 10 and 20 - which googling seems to suggest are 10Gbps and 20Gbps capable.

I have the need to have a large amount of data be periodically written and constantly read for work reasons.
Essentially, for work I use a digital art assets management system that can tier data access such that the system will first look for the data on local drive, then a local network share or equivalent, etc, etc, until finally it hits the work VPN to download it to store locally after exhausting faster "caches".

Currently it�s just my work computer with an Nvme drive. But the drive is only 1TB. And because I�m remote hitting the VPN to download isn�t cutting it for me.

I also wouldn�t mind using storage for non-work purposes such as streaming to my living room AppleTv.

What would be good options for my use case and hardware limitations?
Maybe I would be better off buying something like the Samsung T9 for the work computer and doing NAS for the streaming?
The 4TB version might work but smaller than I�d like. Is there something like that but with 10TB?

This sounds like it's entirely your work's problem. Ask them for a larger internal drive. Don't mix your poo poo with work poo poo. If they won't do it, enjoy your coffee breaks while stuff downloads over the VPN.

# ? Mar 14, 2024 14:49

Wizard of the Deep: Sep 25, 2005; Another productive workday

Henrik Zetterberg posted:

Suggestions for a UPS that can talk to my 8-disk Synology about gracefully shutting down in the event of a power loss?
And I have a NUC right next to it running Win11. I assume the UPS would have Windows software to do the same? Can you USB 2 machines into 1 UPS with control like that?

I don't need something that has like a 6 hours runtime or whatever. Just long enough for a graceful shutdown.

My Synology Just Works� when I plug an APC UPS into the USB port. Control Panel > Hardware & Power > UPS gives control over everything, and it picks up everything included estimated runtime on Device Information.

# ? Mar 14, 2024 15:06

powderific: May 13, 2004; Grimey Drawer

Trying to figure out storage and backup long term for my work stuff, and I'm beginning to think it may be best for me to have two NAS, one at the office and one at home, plus external drives as my storage/backup system vs. something with cloud backup. AWS's deepest storage tier looks doable for me price-wise, but the egress fees are so high if I ever did need to restore I think I would have been better off with a second box.

My question is this: is there a good way to do sync between two locations where one has fast network and the other decidedly does not? I'll have a mix of raw video files that are by far the biggest storage use, and then project files, music, etc. that are much smaller. Each job I do I'd be loading about 1-3 TB of data.

Ideally I'd be able to sync the smaller files automatically over the internet, and then have some way for the systems to mark which bigger files need to be taken to the second location. At this point I'm thinking I'd go with Synology as I'm not sure I want to be going down a truenas rabbit hole.

# ? Mar 14, 2024 16:02

kri kri: Jul 18, 2007

Syncthing has worked well for me, I have 2 remote servers I sync to.

# ? Mar 14, 2024 23:59

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

powderific posted:

AWS's deepest storage tier looks doable for me price-wise, but the egress fees are so high if I ever did need to restore I think I would have been better off with a second box.

I would go with a solid setup for assuring availability of your data locally, making it extremely unlikely you would ever need to restore from your cloud backup - so just the one NAS with appropriate redundancy (raidz2). At the point you need to restore from a cloud backup poo poo will have really hit the fan and you're probably willing to pay whatever it costs to get the data back. There's a lot to be said for the assurance & reliability of a cloud hosted solution that is managed by entire teams of other people, and you only access it via write-only credentials. Sure you could do a great job of this the DIY route with two NASes, maybe it's just me that wouldn't be able to *really* trust my own setup for a real DR scenario though.

Does your cost comparison account for the fact that you will be replacing your drives every X number of years as the warranty expires? There's also the small chance of other type of hardware failures

For the truly irreplaceable stuff, I just can't bring myself to trust a completely DIY solution. Cloud backup using append-only credentials is a must!

# ? Mar 15, 2024 00:18

Shumagorath: Jun 6, 2001

The speculation is that Glacier is using Bluray machines, and that's about as good as you can get for data integrity unless you want to mandate human readability, in which case lol

# ? Mar 15, 2024 04:33

powderific: May 13, 2004; Grimey Drawer

An aspect of my calculation here is that I need to use the files mainly at my office, which has slow internet, so I need some kind of fast bulk storage there regardless and that gives replicating NAS boxes benefits besides just backup. I suppose I could use a smaller DAS array at the office and have a NAS at home backing up to the cloud. In an ideal world I'd use AWS or a tape drive to be my third backup and archive old work that I could live without but don't want to straight up delete.

# ? Mar 15, 2024 05:35

Clark Nova: Jul 18, 2004

Shumagorath posted:

The speculation is that Glacier is using Bluray machines, and that's about as good as you can get for data integrity unless you want to mandate human readability, in which case lol

it's wild that there are commercial data storage services where the underlying technology is "we're not telling you" :iiam:

# ? Mar 15, 2024 17:50

Twerk from Home: Jan 17, 2009; This avatar brought to you by the 'save our dead gay forums' foundation.

Shumagorath posted:

The speculation is that Glacier is using Bluray machines, and that's about as good as you can get for data integrity unless you want to mandate human readability, in which case lol

I can't be arsed to chase down the long tail of rumor articles plus occasional NDA leakage but write-one shingled hard disk has been really cheap for a while.

Also nothing says you have to keep a hard disk powered on.

# ? Mar 15, 2024 17:57

priznat: Jul 7, 2009; Let's get drunk and kiss each other all night.

Plot twist: it is actually all old Zip disks

# ? Mar 15, 2024 20:56

BonoMan: Feb 20, 2002; Jade Ear Joe

priznat posted:

Plot twist: it is actually all old Zip disks

*click*

# ? Mar 15, 2024 22:38

BlankSystemDaemon: Mar 13, 2009

Shumagorath posted:

The speculation is that Glacier is using Bluray machines, and that's about as good as you can get for data integrity unless you want to mandate human readability, in which case lol

Bluray density is dwarfed by tape so it's not that good.
Microsoft is using media that's about the size of an optical disc and can store 7TB - by using nanometer-scale quartz (on a substrate) that's holed like a punchcard using a laser, they're resistant to EMF and still work as an effective WORM media.
Barring extinction-level events that can take out Iron Mountain and places like it, it's probably the best humanity will manage for quite a while.

Twerk from Home posted:

I can't be arsed to chase down the long tail of rumor articles plus occasional NDA leakage but write-one shingled hard disk has been really cheap for a while.

Also nothing says you have to keep a hard disk powered on.

Treating host-managed zoned storage as WORM media was one of the very earliest use-cases I heard about it for, so it makes perfect sense.
Too bad, for us, that they can only be bought by the hyperscalers.

BlankSystemDaemon fucked around with this message at 00:47 on Mar 16, 2024

# ? Mar 16, 2024 00:44

chocolateTHUNDER: Jul 19, 2008; GIVE ME ALL YOUR FREE AGENTS

ALL OF THEM

BlankSystemDaemon posted:

Bluray density is dwarfed by tape so it's not that good.
Microsoft is using media that's about the size of an optical disc and can store 7TB - by using nanometer-scale quartz (on a substrate) that's holed like a punchcard using a laser, they're resistant to EMF and still work as an effective WORM media.
Barring extinction-level events that can take out Iron Mountain and places like it, it's probably the best humanity will manage for quite a while.

Thank you for the link - that is some seriously cool technology. in a weird way, it almost made me...nostalgic? For those "early" days in tech where you you would read about Google trying something cool like this every other day in 2008 or whatever.

# ? Mar 16, 2024 03:21

Azhais: Feb 5, 2007; Switchblade Switcharoo

chocolateTHUNDER posted:

Thank you for the link - that is some seriously cool technology. in a weird way, it almost made me...nostalgic? For those "early" days in tech where you you would read about Google trying something cool like this every other day in 2008 or whatever.

You can still!

https://killedbygoogle.com/

# ? Mar 16, 2024 03:25

Adbot: ADBOT LOVES YOU

# ? Apr 29, 2024 01:49

Yaoi Gagarin: Feb 20, 2014

wait gently caress they're killing podcasts I actually use that

# ? Mar 16, 2024 04:05

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > NAS/Storage Megathread: What is this "File Deletion" You Speak of

«‹›842 »