|
What kind of data does one want to "throw in a closet for a few years" and not touch? You'll want to keep a good record of what files/folders are on what SSDs.
|
# ? Jul 1, 2023 20:15 |
|
|
# ? Jun 3, 2024 22:20 |
|
Corb3t posted:What kind of data does one want to "throw in a closet for a few years" and not touch? You'll want to keep a good record of what files/folders are on what SSDs. I can see it happening for research data - sequencing raw data is huge, and you'll probably never need it after extracting the (comparatively tiny) summary files. But it can make sense to keep them anyway, in case any questions come up about an article you've used the data in, or you think a new tool can pull out more information. It doesn't have to be especially complicated, though: Throwing the files on a stack of tapes with a simple filename/sample index in a text file, and writing "fasta files FGFR1-paper 2023" in sharpie on the boxes? Probably good enough! (Bonus points for an offsite copy in another format, of course.) Computer viking fucked around with this message at 20:42 on Jul 1, 2023 |
# ? Jul 1, 2023 20:40 |
|
Don't have much data actually. Just saw that reddit is closing their API and wanted to grab the data just in case. Not even sure what I would use it for but just the fact they're trying to shut it down gives me the nagging feeling that I want to have it. I think a single snapshot of reddit comments is about 250gb but if I grab monthly snapshots that 3tb per year. If I want to grab ten years that's 30tb. And then I'm multiplying by about 3x just in case I was wrong and underestimated something. If there's a better way to do that, I'd love to hear it. I just want to use my laptop to set it up if possible. Don't want to have to build a PC or buy a monitor since I have no space and I don't want to worry about whether a cloud hosting provider will change their prices, go out of business, etc. oliveoil fucked around with this message at 23:14 on Jul 1, 2023 |
# ? Jul 1, 2023 23:12 |
|
For something like that, a tower PC with a stack of 16TB drives in raid5/raidz seems fine, and then you just pull and archive the drives. You could get one of those 'plop in a 3.5" drive' USB docks and do it one drive at the time, but it's probably useful if you can recover from a dead drive. Alternatives include a bunch of external drives and a powered USB hub, or one of the IcyDock or similar USB JBOD boxes. Just don't use any of the proprietary hardware raid solutions, you'll never get the data out again. Computer viking fucked around with this message at 23:34 on Jul 1, 2023 |
# ? Jul 1, 2023 23:31 |
Computer viking posted:I can see it happening for research data - sequencing raw data is huge, and you'll probably never need it after extracting the (comparatively tiny) summary files. But it can make sense to keep them anyway, in case any questions come up about an article you've used the data in, or you think a new tool can pull out more information. It doesn't have to be especially complicated, though: Throwing the files on a stack of tapes with a simple filename/sample index in a text file, and writing "fasta files FGFR1-paper 2023" in sharpie on the boxes? Probably good enough! I feel seen
|
|
# ? Jul 1, 2023 23:33 |
|
oliveoil posted:Don't have much data actually. Just saw that reddit is closing their API and wanted to grab the data just in case. Not even sure what I would use it for but just the fact they're trying to shut it down gives me the nagging feeling that I want to have it. You also need to consider how fast is your internet connection and how long it would take to transfer 30 TB of data. And how long it would take for Reddit to notice how much bandwidth you are wasting and ban your IP, or slow your download speed to a crawl.
|
# ? Jul 1, 2023 23:44 |
|
at those amounts of data your ISP might get involved too and either want you to buy a business plan or just throw you off
|
# ? Jul 1, 2023 23:46 |
|
I think I’m preferring regular Linux distros and don’t want to run TrueNAS or UnRAID while I’m building my NAS. I’m planning on doing zfs with RAIDZ2. Any issues with running Ubuntu or Arch and running zfs on there?
|
# ? Jul 2, 2023 00:14 |
|
Boner Wad posted:I think I’m preferring regular Linux distros and don’t want to run TrueNAS or UnRAID while I’m building my NAS. I’m planning on doing zfs with RAIDZ2. I would not run a rolling update distro like Arch as a NAS. I'd also go with Debian over Ubuntu but that's just personal preference. Otherwise, if you've got the Linux chips, go hog wild. Maybe do all your config with Ansible.
|
# ? Jul 2, 2023 00:18 |
|
Could I suggest FreeBSD? It's a little bit different, but not really harder to pick up than e.g. Arch. The big benefit here is that ZFS is natively supported, so you can install to and boot from it, and you'll never have issues with the DKMS module not being available for the kernel you just upgraded to.
|
# ? Jul 2, 2023 00:34 |
|
Boner Wad posted:I think I’m preferring regular Linux distros and don’t want to run TrueNAS or UnRAID while I’m building my NAS. I’m planning on doing zfs with RAIDZ2. I have been running zfs on Ubuntu for years now and it's been rock solid.
|
# ? Jul 2, 2023 01:00 |
|
oliveoil posted:Don't have much data actually. Just saw that reddit is closing their API and wanted to grab the data just in case. Not even sure what I would use it for but just the fact they're trying to shut it down gives me the nagging feeling that I want to have it. I don't think that's correct on the size. When the API was cut to the team powering all the archive things I grabbed the last version a single month comments was 31G for 12/2022. All comments and posts from June 2006, through December 2022 was 1.9Terabytes, and it's not like they're adding new content to 6/26. Even at 30% yearly growth over the next 10 years it'd be 4ish terabytes for an entire year of content, and I don't think reddit is going to be growing at 30% year over year. I don't know how you're going to get it now without the API though.
|
# ? Jul 2, 2023 05:10 |
|
Boner Wad posted:I think I’m preferring regular Linux distros and don’t want to run TrueNAS or UnRAID while I’m building my NAS. I’m planning on doing zfs with RAIDZ2. No issues but I suggest proxmox as it's Debian based and has been solid for me since I swapped from ESXI passing hardware to FreeNAS. Proxmox 8 just came out based on Debian 12 (Bookworm)
|
# ? Jul 2, 2023 05:12 |
|
Boner Wad posted:I think I’m preferring regular Linux distros and don’t want to run TrueNAS or UnRAID while I’m building my NAS. I’m planning on doing zfs with RAIDZ2. Openmediavault is just Debian with a very FreeNAS like webUI on top (because it was made by a former FreeNAS dev). It can do zfs.
|
# ? Jul 2, 2023 06:16 |
|
I'm gonna build a nas and a home server after I build my new computer.
|
# ? Jul 2, 2023 19:04 |
|
are there any celeron mini PCs with built in IPMI, or is the pikvm the only real option for something like that
|
# ? Jul 2, 2023 19:59 |
|
e.pilot posted:are there any celeron mini PCs with built in IPMI, or is the pikvm the only real option for something like that https://www.supermicro.com/en/products/system/mini-itx/sys-e200-9b.cfm That model's availability hasn't been amazing for the last year or two so we've been using a prebuilt system from MITXPC based on one of these as an alternative: https://www.supermicro.com/products/motherboard/atom/x10/a1sri-2558f.cfm For the life of me I can't understand why no one sells something along the lines of an AST2500 on a PCIe card. I realize that some integration features like editing BIOS settings from the web UI would not be possible in a standalone card format but the core features most people care about could be implemented with nothing more than a PCIe card and a pigtail for the power/reset header. Maybe a USB pigtail might be more reliable than acting as a USB expansion card, either way it doesn't seem like there are technical barriers to such a thing. Maybe ASpeed just won't sell the chips for that kind of use?
|
# ? Jul 2, 2023 22:04 |
|
wolrah posted:As far as I've been able to find this is the smallest/cheapest PC out there with real remote KVM and remote media support: bleh that’s unfortunate, I guess I’ll stick to the micro server for now
|
# ? Jul 3, 2023 01:48 |
|
Speaking of PiKVMs, is there a KVM solution that works without buying hard to come by Pis but isn’t super expensive?
|
# ? Jul 3, 2023 02:23 |
|
Korean Boomhauer posted:Is there any easy way to tell what stick of ram is throwing constant "Correctable ECC - Asserted" on an asrock rack board? I went from 1 stick of ram to 6 on my board and ran memtest on it before booting into truenas but now i get ECC errors about once a day. I ran memtest on it again and nothing. The IPMI won't tell me what stick it is either. I think i forgot to mention but this board is an Asrock rack. their support isn't terribly helpful either
|
# ? Jul 3, 2023 02:30 |
e.pilot posted:are there any celeron mini PCs with built in IPMI, or is the pikvm the only real option for something like that
|
|
# ? Jul 3, 2023 03:05 |
|
Just to double check, a PCIe-2.0 8x HBA card would be just great in a PCIe-3.0 4x lanes/8x slot? Especially if most of the drives are mechanical anyway.
|
# ? Jul 3, 2023 21:15 |
|
It'll work. What kind of HBA is this?
|
# ? Jul 3, 2023 21:21 |
|
Wibla posted:It'll work. What kind of HBA is this? I have a couple of the reflashed Dell H310's from ebay
|
# ? Jul 3, 2023 21:29 |
|
Chilled Milk posted:Just to double check, a PCIe-2.0 8x HBA card would be just great in a PCIe-3.0 4x lanes/8x slot? Especially if most of the drives are mechanical anyway.
|
# ? Jul 3, 2023 21:47 |
|
Pablo Bluth posted:PCIe 2.0 has 0.5GB/s per lane, so the card at x8 offers 4GB/s. In a 4x lane slot, it'll be limited to 2GB/s. I think achievable write throughput will be lower depending on redundancy (software RAID-10 or ZFS miror has to send the info twice on write, although I stand to be corrected). Whether that sounds like a bottleneck depends on your situation/desires. I see, so yes it'll just run at 2.0 speeds over the available lanes. 2GB/s could be a bottleneck given the mix of drives (5 mech, 3 sata ssd), but barely. Thanks y'all
|
# ? Jul 3, 2023 22:38 |
I'm running into an rsync issue and can't quite figure it out. Here's the command I'm running and the error I'm seeing (running as some_user on home-server, which is a Debian machine).code:
code:
I came across this Rsync: mkstemp failed Operation not permitted thread on the TrueNAS forums that described a similar error, but I didn't have any luck trying the --no-perms option on there. I think the issue is because I am mixing linux & windows file permissions, so perhaps I should be using SMB instead of NFS on home-server? I'm using SMB because I connect to these datasets from a Windows machine (as some_user) as well.
|
|
# ? Jul 4, 2023 02:52 |
If memory serves, maproot only maps I/O by remote root access to a UID. So unless you're running rsync as root on the remote machine, it won't have any effect.
|
|
# ? Jul 4, 2023 08:34 |
BlankSystemDaemon posted:If memory serves, maproot only maps I/O by remote root access to a UID. I tried unsetting the maproot user & group, and set mapall user & group to some_user. Still running into the same error though. From the machine I'm running the rsync command on (home-server) If I run touch /mnt/whatever/lol.txt, it works just fine too. If that works why is rsync complaining about permissions??
|
|
# ? Jul 4, 2023 19:16 |
fletcher posted:I tried unsetting the maproot user & group, and set mapall user & group to some_user. Still running into the same error though. You can also use dtruss if you have sysutils/dtrace-toolkit available; it works the same, but uses dtrace underneath.
|
|
# ? Jul 4, 2023 20:13 |
|
I’m moving over to using nextdns after being a pi.hole user for a while, anyone use it with unraid? I have the router (netgear pos) set up right now but would like to identify the traffic for devices behind it including the unraid. Just wondering if there is a way someone else has discovered to do that, my cursory googling hasn’t turned up a lot.
|
# ? Jul 4, 2023 20:38 |
|
priznat posted:I’m moving over to using nextdns after being a pi.hole user for a while, anyone use it with unraid? I have the router (netgear pos) set up right now but would like to identify the traffic for devices behind it including the unraid. See if you can use the nextdns client on your router, it will pass through names based on your DHCP clients info.
|
# ? Jul 4, 2023 20:48 |
|
Dyscrasia posted:See if you can use the nextdns client on your router, it will pass through names based on your DHCP clients info. Not currently, it’s a fairly locked down netgear but I am thinking about an upgrade at some point so that will be a must have for whatever I do get! It would be useful to know if some IoT gizmo is causing a ton of sketchy requests.
|
# ? Jul 4, 2023 20:51 |
|
priznat posted:I’m moving over to using nextdns after being a pi.hole user for a while, anyone use it with unraid? I have the router (netgear pos) set up right now but would like to identify the traffic for devices behind it including the unraid. If you run a custom forwarding resolver on unraid, you can append arbitrary strings that NextDNS will show you as distinct devices in the UI. https://help.nextdns.io/t/x2h76ay/device-information-log-enrichment Alternatively, if you can make unraid resolve via DoH or DoT directly, you can append the client identifier.
|
# ? Jul 4, 2023 21:23 |
|
Time for another "ZFS has saved my abomination of an array" post.code:
|
# ? Jul 5, 2023 17:27 |
|
Boner Wad posted:Speaking of PiKVMs, is there a KVM solution that works without buying hard to come by Pis but isn’t super expensive? Pi's are becoming much more available now. I just got a zero 2 W, and I got a 4 last week. From legit shops, not scalpers. Just keep rpilocator.com open, or keep checking. Actually, the stores seem to be actually sending availability emails now that they have decent stock. Actually, I might be best to grab the kvm hat bit off Ali express in case they start selling through those now.
|
# ? Jul 5, 2023 22:57 |
|
the new block-clone should make the "two zvol block devices that branch from a common root image (base OS layer or application ecosystem etc)" run wayyy better right? that's always been the thing people are flirting at with ZFS dedup and generally COW layers, you want to build one linux image and have that run with deduped memory, and be able to roll the base image up and dedup and keep as much as you can. Docker notionally has a ZFS backend that could do some of this kind of poo poo apart from some crucial flaw, not sure if this helps because you can create new datasets that are hardlinked at a block level?
|
# ? Jul 6, 2023 08:33 |
|
Paul MaudDib posted:
Last time I used the ZFS backend it was slow as hell. It is faster to just create a ZVOL and format it as EXT4 than running the ZFS backend. I am talking multiple minutes when composing a docker file instead of seconds.
|
# ? Jul 7, 2023 07:56 |
Mr Shiny Pants posted:Last time I used the ZFS backend it was slow as hell. It is faster to just create a ZVOL and format it as EXT4 than running the ZFS backend.
|
|
# ? Jul 7, 2023 10:18 |
|
|
# ? Jun 3, 2024 22:20 |
|
BlankSystemDaemon posted:How can it possibly be that slow unless someone intentionally added a bunch of wait calls?! I have no clue. I just went on with my life and formatted a ZVOL with EXT4. The difference is/was night and day.
|
# ? Jul 7, 2023 16:23 |