Self Hosting: It's like being a prepper for the cloudpocalypse

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Self Hosting: It's like being a prepper for the cloudpocalypse

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Potato Salad posted:

rolling your own CA is so fun though

I use a standalone GUI called XCA to generate an offline root CA and sign a 10-year intermediate that I hand to ADCS to sign everything else. I also have another intermediate off of that for Smallstep CA to do all my ACME poo poo.

It was absolutely exhausting and mind-melting to get everything set up just right, but it's been very hands-off ever since.

And then there's the occasional self-hosted app (NextCloud :argh:

) whose mobile app shits the bed if you don't have a globally trusted cert like LetsEncrypt.

# ¿ Jan 20, 2023 15:54

Adbot: ADBOT LOVES YOU

# ¿ May 14, 2024 04:43

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

corgski posted:

https://dns.he.net

There's no reason to pay for someone to host your nameservers, it's free from Hurricane Electric or a free value-add from just about every registrar these days.

HE.net DNS is great. I would switch back in a heartbeat if they had an API that could do ACME DNS challenges. The only free DNS provider I've found that can do those is Cloudflare, so I'm stuck with them.

# ¿ Jan 24, 2023 20:31

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

corgski posted:

Dynamic TXT records have been added!

Well, there's my new project for the week. Thanks!

# ¿ Jan 24, 2023 22:52

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

I have a Ceph pool in my Proxmox cluster that was on some cheap 2.5" laptop spinners, and it was always "meh" - but that was all the additional storage I could fit in my hosts (an assortment of OptiPlex micros). I never really wanted to put anything on the pool because it was so drat slow, any VMs I put there would inevitably have errors or randomly crash, but I still wanted to have some amount of clustered shared storage just in case.

I switched those out for some Intel DC SSDs and it's actually quite usable now. Those DC-class SSDs are nice, too, because they have multiple PBs of write endurance compared to the 200-300TB consumer NVMe drives. So I'm no longer skittish about putting high-IO stuff on them (InfluxDB, Graylog, etc.)

Proxmox good, SSDs gooder

# ¿ Jan 27, 2023 15:07

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Yeah, that (virtualized NAS) is a setup I've tried and would always advise staying away from. It's just been pure pain for me.

I originally wanted to go that route when I moved to a server that supported HBA passthrough because I hated the idea of my NAS sitting at single-digit utilization all the time. Hardware maintenance and reboots sucked, and my new top fear became "what if my hypervisor dies and takes my NAS with it".

Bare-metal NAS is simpler and one less failure point to worry about. I got around the wasted resources issue by using the NAS as a Docker host, which also shrunk my VM footprint by a fair amount.

# ¿ Jan 28, 2023 06:56

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

BedBuglet posted:

I don't want to see my throughput negatively affected by running things through a proxy.

You won't. HomeAssistant is never going to push enough data that it overwhelms something like Traefik. I have HomeAssistant running on Docker with a Traefik reverse proxy in front to handle the SSL and it's solid. Kubernetes is functionally no different aside from having distributed networking and control plane.

As cumbersome as the setup may seem, the Traefik process itself is very lightweight and the type of routing it does is no more resource-intensive than iptables masquerading, which Kubernetes is already doing under the hood to power the cluster networking.

If you really don't want something sitting in front of HA, you might look into MetalLB which can assign an external IP to a service and use gratuitous ARP to advertise it to the rest of the network. If the service isn't running on the speaker node, it just routes to the correct host through the native K8s cluster networking.

# ¿ Feb 13, 2023 16:19

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

SEKCobra posted:

What is the goto for media streaming to the TV? I've been using UMS on my PC for years to stream to my webOS TV, but I'm starting to think I could probably just move my media onto my NAS and stream via my server. Usually don't need to transcode anyway.

Plex and Jellyfin are the two biggest self-hosted streaming platforms at the moment.

Plex is commercial software, but is still free and very easy to get started with. The mobile apps cost money, but there is a paid tier that also unlocks them. It has better app support across pretty much every mobile and smart TV platform. Plex has a cloud service to enable remote streaming from your server, but it's also capable of running completely isolated, too, so best of both worlds really.

Jellyfin is open source, but is lagging a bit on app support because of that. It's not a bad option, but my experience is that Plex is closer to "just works" in just about every respect.

# ¿ Feb 14, 2023 18:24

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Plex having an exploitable RCE is the least :stonk:

thing about this incident.

quote:

Already smarting from a breach that put partially encrypted login data into a threat actor�s hands, LastPass on Monday said that the same attacker hacked an employee�s home computer and obtained a decrypted vault available to only a handful of company developers.
...
�The threat actor was able to capture the employee�s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer�s LastPass corporate vault.�

An employee with privileged access to customer information put their corporate LastPass vault on their home PC. This same vault held the access keys to their main S3 buckets that store encrypted customer data.

I don't know why nerds insist on letting work stuff anywhere near their personal computers. You do that and you might as well slap an asset tag on it, because they can fire you and seize that poo poo at a moment's notice.

# ¿ Feb 28, 2023 19:16

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Netgate has been doing sketchy poo poo for a while, so this isn't much of a shock. I read up on the beef between pfSense and OPNsense a few years back when I was moving my poo poo from EdgeOS/Vyatta to an x86 mini PC and that situation alone convinced me to go with OPNsense and never look back. OPNsense rocks and the community plugins are pretty good, too.

I'm struggling to think of a single company that's turned an open-source project into a commercial product and not ended up loving over their users at some point.

# ¿ Oct 26, 2023 17:50

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

mawarannahr posted:

Does sqlite count?

Maybe the way to qualify it would be, any company that sells enterprise licenses for an open source project (versus just selling support hours)

Still thrilled to be wrong, SQLite is cool and good.

# ¿ Oct 26, 2023 19:16

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Nitrousoxide posted:

I personally have dropped portainer for any sort of management. I do keep an instance running but only for its visualization tools for container resource usage. I do everything else via terminal or on my gitlab repo which I push via ssh to the server.

Same, I've got maybe 15-20 containers spread between two physical Docker hosts, and another 10 deployed on K3s. Portainer's great to be able to view those all in a single pane of glass, but I have not moved any of my actual Compose management into Portainer for the sake of having full control.

I still haven't put the mental energy into understanding stacks or the whole v3 Compose spec.

# ¿ Jan 8, 2024 20:22

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Barebones Ubuntu or Debian is a solid choice; it's what I use personally, and it's incredibly stable. If you're that concerned about overhead, maybe Alpine.

It's probably just that I have really bad luck, but every time I've tried to use a container-focused OS, it's been Google Reader-ed within a year.

CoreOS :rip:

RancherOS

K3os

# ¿ Jan 10, 2024 15:22

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Nitrousoxide posted:

CoreOS is still around.

Oh drat! That's great to hear. I just remember seeing the news a few years ago that the original had gone EOL and thought they were trying to absorb everything into Atomic Host.

# ¿ Jan 10, 2024 15:37

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Mullvad costs $5/mo and it's great. They shut down the port forwarding, but it still works fine without it.

I tried usenet via Frugal a few years ago and the experience sucked. Indexers are awful, I'm not putting my credit card into some sketchy Russian website and I'm sure as hell not buying loving crypto to pay for the privilege of searching for some obscure Linux distro.

# ¿ Jan 12, 2024 21:52

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Yeah, gently caress off with the gatekeeping poo poo, as someone who enjoys playing with old hardware that's ridiculously under-powered by today's standards (old SPARC stuff, SGIs, etc.) I appreciate and admire when someone wants to put effort into making something useful with limited resources.

And so would Ron Swanson, FYI.

# ¿ Jan 14, 2024 20:27

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

The differing error codes between internal and external makes me think you might have an incorrect routing configuration on whatever�s directing traffic to the containers (Traefik or nginx I�m guessing).

404 would be the equivalent of �I don�t know what you�re asking for� whereas 502 is �I�m trying to send you to that service but it�s not responding.� I�d start by looking at that part of the stack. Also check the NextCloud logs to see if it�s healthy and is getting any traffic, but the problem is likely before that point.

# ¿ Feb 10, 2024 00:08

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Warbird posted:

What is everyone doing for their monitoring stacks? I realized the other day that I had been losing a docker host to OOM errors off and on for a bit now and usually not noticing for a bit because my Kuma instance was on that host as well. I�ve migrated my Grafana/Prometheus/Kuma stuff onto a different host VM and put memory limits on my container stacks, but I need to get an actual factual monitoring setup going.

I really need to get visibility on the containers themselves, the container hosts, and some sort of log aggregation solution. That sounds like �Actually lean and use Grafana� and �Setup Loki� to me. Is that more or less in line?

For system and application monitoring, I use Telegraf feeding to InfluxDB with dashboards and such in Grafana.

For log aggregation I have Graylog being fed from several inputs depending on the source:

rsyslog for Linux hosts and network devices
Winlogbeat for Windows hosts, using the Beats format and managed by Graylog Sidecar
GELF for Docker and Kubernetes workloads, either using fluent-bit on K8s or the native GELF logging driver in the Docker daemon.

# ¿ Feb 18, 2024 01:52

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Warbird posted:

Seems interesting. Any good writeups for that stuff or is it largely self explanatory?

For the most part a lot of these services I�ve expanded into over the years, fiddled with and found opportunities to integrate them.

Graylog can be a bit of a bear because it�s built on top of MongoDB and Elasticsearch, so while the setup process is well documented, you can end up going down a rabbit hole. The official docs are still pretty good though.

You might have better luck starting with InfluxDB and Telegraf since the setup is more streamlined. The official docs are very straightforward, but you can also find a lot of blogs and guides elsewhere.

# ¿ Feb 18, 2024 16:16

Adbot: ADBOT LOVES YOU

# ¿ May 14, 2024 04:43

Cenodoxus: Mar 29, 2012; while [[ true ]] ; do
��pour()
done

Netdata is good. I ran into a weird issue with Netdata in docker-compose, but my use case was probably a little odd to start with.

If you follow the official instructions, they have you set the container's hostname equal to the FQDN of your host. I also had some other containers for various services that were hosted under the host's FQDN via Traefik routes. One of those containers was Uptime Kuma, trying to probe those services underneath the host's FQDN. So when I ran Netdata, Docker's internal DNS fuckery meant that all those DNS queries from Kuma for my host's FQDN were getting resolved to the Netdata container.

It broke my monitoring for a little while until I figured it out. It wasn't a huge deal - I just commented out the hostname line, but that also meant that my Netdata instance saw the hostname as whatever the container ID happened to be, rather than the name of the host.

Cenodoxus fucked around with this message at 01:37 on Feb 22, 2024

# ¿ Feb 22, 2024 01:33

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > Self Hosting: It's like being a prepper for the cloudpocalypse