Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
CopperHound
Feb 14, 2012

Introduction
  • What is self hosting?
    Self hosting basically means being in control of the systems that host your data as opposed to 3rd party providers.
  • Why self host?
    • You don't want to deal with migrating your data when google decides to kill and/or monetize another service
    • You don't trust 3rd parties with your data
    • You are still a believer in a free and open internet
    • You want something that 3rd parties don't offer
    • You have a bad internet connection and the cloud is too slow on your home network
    • You think that you can somehow do a better job than multi billion dollar companies and love the idea of being a computer janitor
  • What can of stuff can I self host?
    If you can think of it, somebody has probably tried implementing it with varying degrees of success:
    • Social media platform with less or more nazis.
    • Photo organization
    • Video streaming
    • File sync
    • Multiplayer spreadsheets Document collaboration
    • Cookbook
    • Home automation (thread)
  • Does self hosting mean a more fractured internet?
    It can, but it doesn't have to. Many things you can self host are stand alone instances that don't talk to others, but some services can talk to each other just like email. Files can be shared across servers. There is an entire federation of social networks.
  • What do I need to get started?
    At the very least, you need a computer with a network connection. Some sort of linux that can run docker will probably be your best choice for starting out. You can self host on all sorts of stuff:
    • a NAS that is holding 18tb of train videos that you have personally recorded (thread)
    • RaspberryPi (thread)
    • A homelab consisting of servers that fell off of a loading dock with fans screaming at 96dBA. (thread)
    • a fanless mini-pc
    • a cloud host
  • I got this great thing running, now I want to access it from the internet
    :frogsiren:HOLD UP BEFORE YOU START FORWARDING PORTS!:frogsiren:
    Network security goes way beyond the scope of this, but only trusted secure services should be accessible from the internet. Things you plan use privately should generally be kept behind a vpn. Things you intend to have accessible from the internet generally should be behind a reverse proxy that handles https encryption. If you know how to set that up, great! If not, there is a docker image called SWAG that can help by packaging an nginx server that automatically generates and renews ssl certificates.

Cool stuff to host:
You can find a great list of stuff here. It can be overwhelming to go through that full list, so here are a list of stuff I and fellow goons have had good experience with:
  • Media Streaming
    • PLEX (thread)
      This is essentially netflix for your own media and has become the standard that all other self hosted streaming is compared to. It is easy to use and you can share your library with friends. It is not a free open source project and some people speculate that most of the recent development has been to build a platform that can be more readily monetized vs working on things the current users actually want.
    • Jellyfin
      Jellyfin seems to be a good open source alternative to PLEX. It was forked from Emby when it went closed source. The web player seems good, but app support might not be as great.
    • Overseerr

      Gay Retard posted:

      Mobile-friendly Plex requesting service. 90% of the time I don't even touch Radarr or Sonnar anymore because its so fast at adding requests.
    • Requestrr

      Gay Retard posted:

      A Discord bot that lets users type "/request ______" to make requests for Plex media.
  • File Syncronization
    • Syncthing
      Syncthing is a cool bit of software. I think of it as more of a peer to peer based cloud sync. You choose what folders you want to share with which computers. They will track changes and keep each other up to date. You can do two-way, upload only, or download only sync. Each computer can choose it's own settings about retaining file versions. This does not support virtual files, so it is not a great option for large amounts of data with small drives.

      VelociBacon posted:

      I wanted to mention that if you're not familiar with this stuff and it seems too complex, I used syncthing to archive/backup photos and videos from my phone onto my server, it was very easy to do.
    • Nextcloud
      Nextcloud functionally is a great self hosted replacement for DropBox. The sync desktop client works as expected and supports virtual files, so you don't have to download everything from the server. It has good file sharing functionality with permissions for public web links, password protection, sharing with specific users or groups, and even sharing with users from other nextcloud servers. There is a lot more to next cloud than just file sync that could have it's own entire post. It has apps that feel like they have the potential to make nextcloud a replacement to google docs or microsoft 365, but right now it falls a little short. The photo management app chokes with more than a few hundred images. You can use collaborative file editing, but it lacks the access permission option for comment & suggest changes only. Despite these shortcomings, the cookbook app is A+ and makes the entire install worth it.
  • Document Collaboration
    • Collabora Online
      LibreOffice in your web browser with simultaneous document editing. It needs to be integrated with another application like nextcloud that handles the files and permissions. It does all the processing on the server, so don't expect to serve several simultaneous users with a RaspberyPi.
    • Onlyoffice
      Only office is another document editor suite that does collaborative editing and comments. Like collabora, it integrates with nextcloud (and more), but this does most of the heavy lifting on the client side instead of on the server.
  • Photos
    • PhotoPrism
      Photoprism is for organizing your photos. It has facial recognition, auto tagging and a map browser. Right now it only supports a single user and albums are shared with secret links that can be set to expire. I would be excited if there was a way to integrate this with nextcloud for the photo library storage and sharing.
  • Utilities
    • Pi-Hole
      DNS based ad blocking.
    • Adguard Home

      Gay Retard posted:

      I recently switched from PiHole to Adguard, and found the overall process a little less cumbersome - mainly the fact that you don't have manually add Block Lists. I've had a lot less issues with random URL forwarders not working as well.
    • Uptime Kuma

      Gay Retard posted:

      An uptime monitoring dashboard for all of your self hosted web applications.
    • nginx Proxy Manager

      Gay Retard posted:

      Easy reverse proxy tunneling with your own custom domain
    • Vaultwarden

      bobfather posted:

      Vaultwarden is an excellent password manager that is about as feature-rich as 1Password but is open source and free. It is based on Bitwarden, who offer free apps for iOS, Android, Windows, MacOS, and browser extensions for all major browsers.
  • Homepage/Web App Portal
What's missing?
Do you have any suggestions for things that I missed or alternatives that might be better?
Maybe something to save people from a screen full of docker?

CopperHound fucked around with this message at 20:59 on Nov 17, 2021

Adbot
ADBOT LOVES YOU

CopperHound
Feb 14, 2012

Somebody who can't shut the gently caress up about BSD posted:

That being said, there's still a benefit to not using the turn-key solutions, which is that they're automatically a much larger target, as these kinds of web-facing software stacks are typically what's being explored by the people who're interested in taking over infrastructure and using it for their own - whether it be to mine crypto (relatively benign, even if not good), send spam (a problem if you ever want to host a mail server, mostly - because of blacklists), host all sorts of bad things (anything from virus to the sort of things that can land you in jail), or any combination thereof.
Which is not to say that it can't also be used by spearphishers or targeted attacks, but that's true for anything.

So at the very least I hope that the people who're reading the thread also make sure that they know how to keep something up-to-date, and ensure that it is kept up-to-date. My experience tells me it's not something one should always take for granted.
Maybe it's even something that can be mentioned in the OP?
Okay, Let's talk about security:
Once you start opening up services for access from the internet, you can no longer bury your head in the sand and assume you are not a target. Everything accessible from the internet will get probed for vulnerabilities. Sometimes it will be something as simple as brute forcing passwords against many of these services that don't have any sort of fail2ban implementation. Other times it will be exploits targeting your specific software or common libraries. Hell, it could even be something like an image file upload allowing remote code execution.

You can mitigate, but not eliminate these threats. This is the bare minimum of what you should be aware of and do:
First you should limit the targets. Do you really need to have sonarr accessible from the web? Even not counting potential undocumented exploits, you are exposing a service that has the ability to delete your media library.
Second, isolate services to only have access to the resources they need. This is where virtualization and containerization is your friend. If you get pwned, you want to limit the damage that can be done. I can't personally speak on much more than data isolation, but network firewalling between services that don't depend on each is a worthwhile consideration that I hope somebody can educate us about. Actual VMs do offer more isolation than containers, but there is the extra overhead of an OS for each vm:

Third, harden your system by keeping everything up to date with security updates.

CopperHound fucked around with this message at 21:41 on Nov 17, 2021

CopperHound
Feb 14, 2012

I would be content if we could just get a timeline scroll in nextcloud. The infinite scroll as implemented just isn't usable.

Gay Retard posted:


Overseerr - Mobile-friendly Plex requesting service. 90% of the time I don't even touch Radarr or Sonnar anymore because its so fast at adding requests.
I'll take a look at this one. I tried out Ombi, but it only sort of functioned. Sometimes.

CopperHound fucked around with this message at 18:52 on Nov 15, 2021

CopperHound
Feb 14, 2012

Karate Bastard posted:

One billion features and none of them work. Or has it gotten better?
The core function of playing mainstream movies and shows is very usable. If you deviate from that the customization is poorly documented or non existent.
The player is still bad if you are using anything other than a 16:9 screen. The client defaults to 2mbit remote playback, so non technical friends will be forcing your server to transcode your blueray rip down to 480p to play on their 4k screen.
Despite all my complaints, PLEX just works without me janitoring it.

I am curious to learn more about music streaming alternatives. I feel like YTMusic and Spotify always steer me to the same 20 or so songs despite having the world of music available. Does your music streaming set up play nicely with carplay or Android auto apps?

CopperHound
Feb 14, 2012

Slow is Fast posted:

So I've been wanting to replace google photos since they're complaining about my storage usage.
For a server head I have an intel atom z8350 1.44ghz with 4 gigs of ram. It already has windows 10 on it and it shits itself over RDP, doing anything, and trying to install docker.
You should be able to use that system to run photoprism. Test it out with a small library of a few hundred photos so you can see if you like it.

It front loads the big processing when it makes thumbnails and indexes with tenserflow, so you might be waiting several hours/days when you dump your full library into it. It should run fine once all that is done.

E: If you want to get several things running at once, it might be worth looking into the home lab hardware. Right now I'm doing everything on my unraid system and I have brought the system to a near standstill a few times running out of memory with 20 Docker containers doing stuff all at once.



Do any of you folx have experience with tools to help with organizing containers running on multiple pieces of hardware?

CopperHound fucked around with this message at 18:00 on Nov 16, 2021

CopperHound
Feb 14, 2012

For some reason I thought photoprism had a quick and easy timeline to scroll through. Am I mixing it up with the very much unfinished librephotos?

CopperHound fucked around with this message at 18:44 on Nov 16, 2021

CopperHound
Feb 14, 2012

Nitrousoxide posted:

Does photoprism let you share a photo with people as a link?


This is from an install I haven't set up with nginx yet, but I imagine that localhost should get replaced by whatever your public address is.

CopperHound
Feb 14, 2012

BlankSystemDaemon posted:

Maybe it's even something that can be mentioned in the OP?
Okay, I added something of a don't be stupid list of a minimum to do.

CopperHound
Feb 14, 2012

Any of y'all have a certain way you like to implement DNS to local non routable ips? Do you use a valid global tld like server.local.plsdonotpwnme.com or something like myserver.lan?

CopperHound
Feb 14, 2012

FireTora posted:

Yup, works with a domain for albums. If you want to share just one photo then copying the link to the image has the api access baked into the URL so it van be viewed without logging in.
I was looking at this and for the life of me I can't find a way to revoke access to individual photo links like you can with albums.

CopperHound
Feb 14, 2012

I'm looking into authentik to set up single sign on. It looks promising, but I can't wrap my brain around how to make it work. The documentation very much feels like it was written by the person doing the programming.

I'm trying to follow the directions for nextcloud step by step, but I'm getting an error about the user not being provisioned.

It probably doesn't help that I don't know the difference between my rear end and LDAP.

CopperHound
Feb 14, 2012

Matt Zerella posted:

I've been downloading sample packs recently (bought legally) for futzing around with in Ableton and a few samplers.

I'm having the damndest time finding this but is there some kind of nice web gui I can feed a URL to and tell it where to download to? Preferably one with a nice adaptive webui that I can use on my phone or iPad?

This would be on UnRAID so a docker is preferred.
I haven't tried, but aria2+webui might be what you're looking for.
https://forums.unraid.net/topic/60117-support-aria2-webui/

CopperHound
Feb 14, 2012

Manually forwarding the port with upnp disabled is the more secure option. The functionality will be the same without giving malware on your lan an easy way to punch through your firewall.

CopperHound fucked around with this message at 17:56 on Dec 5, 2021

CopperHound
Feb 14, 2012

I've been spending the past week or two giving myself a crash course on self hosted kubernetes clusters. I just barely got the self contained ha control plane and load balancer figured out along with basic ingress with traefik. If I find it at all practical for home use I'll try posting a guide to get some basic stuff hosted.


Holy hell this is a steep learning curve compared to unraid docker containers.

CopperHound
Feb 14, 2012

Matt Zerella posted:

Are you doing this for learning or have you watched too much TechnoTim on YouTube?
What got this started was me wanting to have a dhcp server integrated with local DNS so I could just type whateverhostname.local.mydomain and my current router doesn't support that. I also want some fault tolerance, so one computer going down doesn't break my whole network.

I probably should just figure out how to do failover with pihole.

CopperHound
Feb 14, 2012

Matt Zerella posted:

If you want "easy mode" for kubernetes, look into Rancher.
Sure was hard for me to figure out how to get easy mode up and running, but now I got that far. I think my next step is to wrap my brain around longhorn then kompose.

CopperHound
Feb 14, 2012

Hot new zero day exploit dropped. If you're running anything Java based, check out if you are vulnerable. https://www.docker.com/blog/apache-log4j-2-cve-2021-44228/

Wouldn't want y'all to get pwned because you still like Minecraft.

CopperHound
Feb 14, 2012

I haven't taken a moment to try it out yet, but this looks promising for downloading podcasts: https://github.com/akhilrex/podgrab

As for getting it from there to your phone I'm not so sure. Have you tried listening to the podcasts with Chronicle ( https://play.google.com/store/apps/details?id=io.github.mattpvaughn.chronicle&hl=en_US&gl=US ) yet?

CopperHound
Feb 14, 2012

I don't know of any ready made solution, but it sounds like you essentially need a front end for a database.

I haven't had a chance to play with it but https://www.nubuilder.com/ might give you the tools to do what you need.

For the database schema I would personally try setting up a relational database like this:
Item Table: Include name of item, and any other defining things you would want to search on.
Location Table: can just be one column of location names, or can go into more detail.
Many to many relationship table: one column to reference item, one column to reference location, and one column for quantity.

E: you could do a self referencing many to one schema if you want to track nested containers, but in that instance I would treat every item as unique instead of having an associated qty.

CopperHound fucked around with this message at 21:52 on Jan 14, 2022

CopperHound
Feb 14, 2012

You can mount a specific file instead of directory with docker compose. Could that be a solution for you?

CopperHound
Feb 14, 2012

fletcher posted:

Anything else I'm not thinking of? Is there anything that I'm going to really miss after this switch?
I think the thing you might miss is just how much google integrates sharing of everything with one sign on.

It is possible to use a single sign on front end with nextcloud, but photoprism is not multiuser and only supports link sharing.

The SSO stuff is hard for me to comprehend, but I did get allauth working for nextcloud.

e: I take that back it was "Authentik"

CopperHound fucked around with this message at 02:50 on May 12, 2022

CopperHound
Feb 14, 2012

Billy Ray Blowjob posted:

I'd just like to acknowledge how everyone who says Docker and containers are easy, and how in real life its worse than using Linux in 2004.
What we mean to say is that it is easier than trying to have all services running simultaneously natively on bare metal or devoting a VM to each one.

My first real experience with docker was with unRAID and the community apps plugin. It made docker nearly as simple as an app store.

As for compose or command line, many docker hub pages tell you exactly how to start them with compose or the command line.

If there isn't an official image, I usually go for something from linuxserver.io for the sake of consistency.

E: one problem I occasionally have is with file permissions on mapped directories. I usually manage to fix it but it seems super inconsistent.

CopperHound fucked around with this message at 05:19 on May 13, 2022

CopperHound
Feb 14, 2012

I would say optimizing databases is beyond the scope of personal use. Do a DB for each application and avoid accidentally loving up backups or permissions.

Idk, if you want to have each instance in its own container or not tho.

CopperHound
Feb 14, 2012

Thanks for sharing, that's it pretty cool that I can pull multiple sources together.

CopperHound
Feb 14, 2012

I spent the last 6 months trying out proxmox and decided it was only making the bullshit I wanted to run at home more complicated. I had it running on my nas and two usff PCs, and these are the extra things I struggled with:

  • Nodes of the cluster showing unknown status
  • Containers refusing to start
  • Containers refusing to stop
  • Updating the OS and learning the joys of ballooning log files for each vm and lxc container
  • Trying to find the configuration files or commands to do things not supported in the web interface like bind mounts.
  • Container start up order only applies per node and not across cluster

On the plus side, backups were simple and I managed to do all my fumbling around and transition away without any data loss.

I will admit, in retrospect, I was doing some stuff that seems quite dumb:
I wanted to run some drives with mergerFS and SnapRaid, so I passed them through to an open media vault VM and shared them back to the cluster with NFS. A lot of my struggles seemed to relate to me not running the NAS bare metal.

CopperHound
Feb 14, 2012

spincube posted:

I've found a nice Android app that can stream music from a few self-hosted applications: https://symfonium.app It's paid, with a full seven-day trial.

In my experience so far it plays really, really well with a Navidrome install, and offers a few choice features on top: like smart playlists, and falling back to an offline cache, and I appreciate being able to remove parts of the UI that I'll never use.
I have been using this for a bit now and need to ask is there an app like this for desktop?

CopperHound
Feb 14, 2012

Well Played Mauer posted:

I need to get another 3.5->2.5 conversion kit.
That is a funny term for cramming a drive wherever it fits with double sided tape.

CopperHound
Feb 14, 2012

cruft posted:

I don't suppose anybody in here knows whether btrfs mirroring (as opposed to btrfs on mdraid level 1) is something OpenMediaVault can handle?

If that isn't the nerdiest thing I've ever posted, I don't know what is.
Yes, in the sense that it is debian, but you might need to use the command line.

The web UI will see the file system once you have it setup

CopperHound
Feb 14, 2012

Warbird posted:

Back to email talk, I’ve been half thinking about getting a domain to have a fancy dumb email address instead of a @gmail one. Is this a bad/dumb idea for personal use? I could see problems arising if the domain price spiked or something. Afaik you can’t just outright own anything anymore so that seem like a potential nonstarter.
A lot of domains are discounted for the first year, just look up the renewal price before picking one. Other than that I have never heard of prices going way up.

There are other options, but simplest thing to do would be to buy a domain somewhere then pay google $5/month to host the email on it.

E: That option gets pricey if you want several inboxes.

CopperHound
Feb 14, 2012

The cookbook app is good. That is all.

Adbot
ADBOT LOVES YOU

CopperHound
Feb 14, 2012

TACD posted:

I want to finally get around to putting together a home NAS so I’m not constantly running out of space for all my downloaded Linux boot discs. I already have a small pile of hard drives floating around from various old PCs, but they’re all different sizes (as in data), some are solid state, some are spinning rust; is there any type of thing I can buy and just plug all my drives in, and have it configure them all into a single RAID-like volume? Or any other recommendations for how to best make use of assorted (but perfectly functional) hard drives?
unRAID is not raid and doesn't have anywhere near the level of data integrity of something like ZFS, but I imagine it might still be exactly what you want.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply