I'm hosting few hobbyist public websites. One is the library of hymns, songs, ... - mostly PDF, MP3, MIDI.
OpnSense router -> Traefik -> Docker container.
That Docker container runs on NAS, which is connected via 1Gbps port.
Suddenly, last week I started to get over 1000 RPS. Was it scanner, parser, crawler, DDOS? I can't tell, as user agent was default Mac OS.
It was using ~1Gpbs bandwidth. From many IPs.
If I would have it hosted at AWS / Azure, it would cost me > $40 / hour for the bandwidth. Unless I use some of their CDN.
My router CPU went high, but it handled it ok.
NAS CPU went high, but it handled it ok.
Traefik loaded the server heavily: it had to both receive and send 1Gbps. And write access logs for all requests. My Alloy & Loki logging system got loaded heavily.
I can relate to the TLS termination issue. It is difficult to provide a self hosted version of a web app with a sensible TLS setup.
The best is almost not to do it at all. Just have a docker image serve http/1.1 and document that a reverse proxy is required to expose the service.
There are simply too many ways to do it and every sys admin/hobbyist has their own preferred way.
For the anonymous and authenticated caching issues the author goes into, I think once again it is useless for self hosting. Just embed a performant web server like nginx for example that is pre configured to serve static files. Use proper headers, and predictable/simple paths. Self-hosted versions are going to be low traffic and just need to work.
Have advanced settings for more control, but keep the default simple and working out of the box with minimal dependencies.
You say caddy. The next person will say nginx-proxy-manager. The next will say DNS challenge let's encrypt behind wireguard VPN. The next will say Traefik. The next will say CloudFlare tunnels.
Everyone has their preferred solution and its always the best and simplest.
Do you want a caddy service embedded with every web app you self host or are you running a single one serving all your apps?
From my understanding if you want acme http challenges to just work with let's encrypt you probably need to use a single, separated caddy to front all your apps.
In that case if I ship a caddy service pre configured to do TLS termination with my app it's just more trouble for you no?
Fair enough, http is enough - I anyways have one caddy to rule them all. However, if your app is picky about reverse proxying, e.g. headers, please add a sample caddy config. It's often the first thing I'm looking for if I'm trying out a tool and want to host it under a subdirectory before configuring a subdomain.
At hobby scale, you don't have to care about the details. Just ship a zip or container image with docs and let the user figure out the rest. Self hosting is going to require some bit of expertise.
I personally don't like apps that ship the entire sink (complex stack) when I already have an opinionated setup that already includes TLS/access control/etc.
I'm glad unencrypted http/1.1 was deprecated. I want a privacy online. TLS also gives you much better data integrity guarantees. Some ISPs - particularly in the US - actively inject javascript code into websites served over plain HTTP.
I just wish nginx / caddy / etc had better out-of-the-box support for letsencrypt. It would be way easier to set up if it was just built in to nginx. Let me specify my domain in my nginx config, and just make it work.
It's partially about how complicated modern Web services can be, and why so many people just give up and ship a Docker container. It's also about how agonizing over the details is sometimes just wasted because all of your users are just AI scrapers.
An art professor I knew used to say about overwrought paintings "too much sugar for a dime". Technical people are often prone to this as well.
Such defeatism indeed. I understand that things look complex but plenty of very complex apps can be self-hosted and they Just Work [TM]. Now, sure, there's work involved. But, just like many others, I can run Plex, JellyFin or Immich just fine and those are more complex than 99% of all webapps ever shipped.
We'got all the world's knowledge compressed and queryable with LLMs: there are way less configuration issues that I cannot solve today than used to be the case.
Did those in charge of those standards royally fuck up HTTPS/TLS? Of freakin' course they did. The interests of the big players, not the small ones, have been taken care of. And the result is an ultra complex mess of fucktarded standards.
But it is what is: it's what we have. And we'll make it work.
I host a webserver for a living, so I might be too far removed from the problem; I don't see anything that 1 minute of effort as part of a less-than-10-minute session with an agentic AI tool can't solve safely. A decent README and project structure is all Codex/CC/Pi needs. I mean this even for the least-technical person who would be self-hosting something.
I guess that breaks the plug-and-play nature, but you can't expect something internet-facing to be plug-and-play. (Maybe that's the real travesty here.)
> Deployments not using Docker will no longer be supported.
This is the way (while also allowing people to use their own proxies etc., but being clear about those not being officially supported).
Ship your back end container, your front end container, tell people how to set up the off the shelf DB containers or whatever else your software needs and if you don’t need 20 different components like Sentry self-hosted versions do, you’ll be fine. The DB can be a container or not. The reverse proxy can be a container or not. Hell, if someone wants to build the back end or front end from source and serve them differently they can, just on their own time. That’s pretty close to the 12 Factor App principles.
Just avoid a lot of complexity when possible: do traditional SSR, or with some progressive enhancements or what I like to do in this day and age (unless maximum accessibility is paramount) only use your back end for the API and have a traditional SPA, none of that complex Next.js stuff. Your front end just becomes a bundle of files that you can put in a web server container to be proxied by whatever the ingress is. At most the Docker entry point would change some values in config.json that the app loads on startup in the browser.
If someone asks for Kubernetes support or whatever, just wish them the best of luck in setting it up themselves!
> you have enough interactivity to warrant a single-page-application as your frontend. however, you also want people without javascript enabled to be able to read the bits that don’t require interaction.
I think at this point it's not a hobby, it's a burden.
I literally just sent some friends a docker image and told them to point Luna at it, for which they were set up in ~20minutes and $3 of tokens, so I'm skeptical.
My homelab is also currently ~40 nix service VM's plus a handful of Ubuntu, etc long tail, and some external DO droplets for NAT, and a more robust vlan config than I'd ever been able to put time towards before... which I haven't had to do more than a dozen hours of manual setup on IN TOTAL thanks to Fable, Sol and local Qwen.
There's truth in this post, but I doubt it's longevity.
They're not all tooootally atomic, I have clustered things like libation+audiobookshelf, ollama+ds4+litellm, but I like that I can manage things almost entirely at the network and host level, generally atomically per service, and especially between services (eg homeassistant and the local ai).
It's DEFINITIVELY overkill, and fills up fast, but nix making everything near fully declarative to roll fwd/back or shuffle between proxmox hosts at any time makes for a clean and easy mental model. Both for me and the AI, because it can infinitely retry a nix flake config until it gets it perfect.
These all seem useful warnings to me, I'm vibing something for hobby scale. The installation script was the one place I needed to reorganize and make a lot of human-powered changes. And I haven't even distributed it yet :)
This is only tangentially related, but I have a love-hate relationship with Immich. Love for everything, but loads of hate for the bug-o-feature that disallows sharing of albums if the website is not hosted over HTTPS.
I can fiddle with the page content in Developer Tools to make it work, but frankly, WTF, especially since Immich expects people to set up a reverse proxy for HTTPS access on their own and does not help them in any way (like providing a pre-configured caddy in their compose.yaml, I don't know)
Honestly, this is a problem that has been solved by LLMs. Just point Claude Code to the server over SSH and say "We need Immich to be served over HTTPS".
You could theoretically upstream the fixed compose.yaml if you felt like it, but the project seemed unusually hostile towards the one fix I tried to upstream.
All of those requirements seem weird. I selfhost a few software and the user count is one. My extent at sysadmin is to write a systemd service, configure the proxy (if it’s on a vps and I use subdomains), and maybe use ansible for scripting the provisioning and deploy.
It’s fine to publish recommendations how to tune configurations for specific usages. But I only need the app, not the various things that you may think I need. And I really want the app to only solve its essential problem. It should not take care of the IT part other than providing configuration samples.
And I do `docker compose up` now with a caddy server in front of docker, so I can just add a new site (sub domain or directory) that reverse proxies to that docker container's port. And the beauty is, that I have a skill for it for claude code, so I actually don't have to do any of that for my local self-hosted stuff.
I just call /publish-local-docker subdomain.mydomain.local
And if there isn't a docker compose, it will create one, and then run it, and update caddy server on its own.
It feels like there could be an entire longform piece on how "modern" web development norms arose because hobbyist/student web developers learned everything in an ad-hoc, local environment without access to, you know, a development server or a mature software stack.
Perhaps the article's title should be, "The web has become a hostile, largely automated environment and naive hobbyists may find it difficult to coexist with professional scrapers, scammers, grifters, bots, hucksters, and cybercriminals."
I have 5Gbps fiber at home. LAN is mostly 10Gbps.
I'm hosting few hobbyist public websites. One is the library of hymns, songs, ... - mostly PDF, MP3, MIDI.
OpnSense router -> Traefik -> Docker container.
That Docker container runs on NAS, which is connected via 1Gbps port.
Suddenly, last week I started to get over 1000 RPS. Was it scanner, parser, crawler, DDOS? I can't tell, as user agent was default Mac OS.
It was using ~1Gpbs bandwidth. From many IPs. If I would have it hosted at AWS / Azure, it would cost me > $40 / hour for the bandwidth. Unless I use some of their CDN.
My router CPU went high, but it handled it ok. NAS CPU went high, but it handled it ok. Traefik loaded the server heavily: it had to both receive and send 1Gbps. And write access logs for all requests. My Alloy & Loki logging system got loaded heavily.
NAS already have cache. I tried to enable cache on Traefik. It went OOM in 2 minutes: https://github.com/darkweak/souin/issues/844
When I searched, I found that Traefik cache with proper memory handling is available only in Enterprise version.
Solved the issue by putting the site behind Cloudflare proxy and enabling it's magic to protect from the bot.
I can relate to the TLS termination issue. It is difficult to provide a self hosted version of a web app with a sensible TLS setup.
The best is almost not to do it at all. Just have a docker image serve http/1.1 and document that a reverse proxy is required to expose the service.
There are simply too many ways to do it and every sys admin/hobbyist has their own preferred way.
For the anonymous and authenticated caching issues the author goes into, I think once again it is useless for self hosting. Just embed a performant web server like nginx for example that is pre configured to serve static files. Use proper headers, and predictable/simple paths. Self-hosted versions are going to be low traffic and just need to work.
Have advanced settings for more control, but keep the default simple and working out of the box with minimal dependencies.
I don't know what you're on about. Just run Caddy and it does it all for you.
Nah he's entirely right. Serve http, let the user do https.
If you want to put caddy in front of yours? Great it works.
For me, running a much larger setup? Great, it also works.
For users who never expose it beyond an IP address on lan? Great it also works.
That is exactly what I am talking about.
You say caddy. The next person will say nginx-proxy-manager. The next will say DNS challenge let's encrypt behind wireguard VPN. The next will say Traefik. The next will say CloudFlare tunnels.
Everyone has their preferred solution and its always the best and simplest.
As a one-person-scale selfhoster that has wrangled Apache and nginx before, I'm also saying go for caddy straight away.
As the next person on this post, I second Caddy
Ok I'll bite again.
Do you want a caddy service embedded with every web app you self host or are you running a single one serving all your apps?
From my understanding if you want acme http challenges to just work with let's encrypt you probably need to use a single, separated caddy to front all your apps.
In that case if I ship a caddy service pre configured to do TLS termination with my app it's just more trouble for you no?
Fair enough, http is enough - I anyways have one caddy to rule them all. However, if your app is picky about reverse proxying, e.g. headers, please add a sample caddy config. It's often the first thing I'm looking for if I'm trying out a tool and want to host it under a subdirectory before configuring a subdomain.
Just provide a docker compose file.
Most of these problems are actually solved with containers, though.
An opinionated release of k8s (k3s, microk8s, etc) does wonders for making much of it just disappear.
You pay a slightly higher up front cost in complexity, in return for largely removing absolutely all the listed problems in this blog.
If you're going to run more than just one app... It's the route I'd recommend.
I think the market is already moving here, too. Synology is exposing very easy interfaces to run container workloads on their devices.
I think we're headed towards something like a combo NAS/router that handles storage and networking (by far the hardest parts of hosting).
> You document what needs to be reverse proxied and give examples for caddy and nginx, repeatedly cursing at nginx for not having an if that’s useful.
Somebody hasn't read the docs and doesn't understand how nginx's "if" works, because it's fantastically useful.
At hobby scale, you don't have to care about the details. Just ship a zip or container image with docs and let the user figure out the rest. Self hosting is going to require some bit of expertise.
I personally don't like apps that ship the entire sink (complex stack) when I already have an opinionated setup that already includes TLS/access control/etc.
I'm not sure I understand this post. Is this talking about self hosting Caddy in a home lab setting or just static web hosting on a VPS?
I think the author is distributing a web app that is run by hobbyists with a lot of esoteric home labs?
I think the author missed the root cause, which is the demise of (unencrypted) HTTP/1.1 connections.
I'm glad unencrypted http/1.1 was deprecated. I want a privacy online. TLS also gives you much better data integrity guarantees. Some ISPs - particularly in the US - actively inject javascript code into websites served over plain HTTP.
I just wish nginx / caddy / etc had better out-of-the-box support for letsencrypt. It would be way easier to set up if it was just built in to nginx. Let me specify my domain in my nginx config, and just make it work.
It's partially about how complicated modern Web services can be, and why so many people just give up and ship a Docker container. It's also about how agonizing over the details is sometimes just wasted because all of your users are just AI scrapers.
An art professor I knew used to say about overwrought paintings "too much sugar for a dime". Technical people are often prone to this as well.
Thank you. This makes more sense. I have an open source web app and it uses Docker (Kamal specifically) to make it easy for others to deploy.
Was this written by the Department of Demoralization?
Such defeatism indeed. I understand that things look complex but plenty of very complex apps can be self-hosted and they Just Work [TM]. Now, sure, there's work involved. But, just like many others, I can run Plex, JellyFin or Immich just fine and those are more complex than 99% of all webapps ever shipped.
We'got all the world's knowledge compressed and queryable with LLMs: there are way less configuration issues that I cannot solve today than used to be the case.
Did those in charge of those standards royally fuck up HTTPS/TLS? Of freakin' course they did. The interests of the big players, not the small ones, have been taken care of. And the result is an ultra complex mess of fucktarded standards.
But it is what is: it's what we have. And we'll make it work.
I host a webserver for a living, so I might be too far removed from the problem; I don't see anything that 1 minute of effort as part of a less-than-10-minute session with an agentic AI tool can't solve safely. A decent README and project structure is all Codex/CC/Pi needs. I mean this even for the least-technical person who would be self-hosting something.
I guess that breaks the plug-and-play nature, but you can't expect something internet-facing to be plug-and-play. (Maybe that's the real travesty here.)
> Deployments not using Docker will no longer be supported.
This is the way (while also allowing people to use their own proxies etc., but being clear about those not being officially supported).
Ship your back end container, your front end container, tell people how to set up the off the shelf DB containers or whatever else your software needs and if you don’t need 20 different components like Sentry self-hosted versions do, you’ll be fine. The DB can be a container or not. The reverse proxy can be a container or not. Hell, if someone wants to build the back end or front end from source and serve them differently they can, just on their own time. That’s pretty close to the 12 Factor App principles.
Just avoid a lot of complexity when possible: do traditional SSR, or with some progressive enhancements or what I like to do in this day and age (unless maximum accessibility is paramount) only use your back end for the API and have a traditional SPA, none of that complex Next.js stuff. Your front end just becomes a bundle of files that you can put in a web server container to be proxied by whatever the ingress is. At most the Docker entry point would change some values in config.json that the app loads on startup in the browser.
If someone asks for Kubernetes support or whatever, just wish them the best of luck in setting it up themselves!
> you have enough interactivity to warrant a single-page-application as your frontend. however, you also want people without javascript enabled to be able to read the bits that don’t require interaction.
I think at this point it's not a hobby, it's a burden.
I literally just sent some friends a docker image and told them to point Luna at it, for which they were set up in ~20minutes and $3 of tokens, so I'm skeptical.
My homelab is also currently ~40 nix service VM's plus a handful of Ubuntu, etc long tail, and some external DO droplets for NAT, and a more robust vlan config than I'd ever been able to put time towards before... which I haven't had to do more than a dozen hours of manual setup on IN TOTAL thanks to Fable, Sol and local Qwen.
There's truth in this post, but I doubt it's longevity.
Why split the nix into so many VMs instead of just systemd units?
They're not all tooootally atomic, I have clustered things like libation+audiobookshelf, ollama+ds4+litellm, but I like that I can manage things almost entirely at the network and host level, generally atomically per service, and especially between services (eg homeassistant and the local ai).
It's DEFINITIVELY overkill, and fills up fast, but nix making everything near fully declarative to roll fwd/back or shuffle between proxmox hosts at any time makes for a clean and easy mental model. Both for me and the AI, because it can infinitely retry a nix flake config until it gets it perfect.
These all seem useful warnings to me, I'm vibing something for hobby scale. The installation script was the one place I needed to reorganize and make a lot of human-powered changes. And I haven't even distributed it yet :)
Damn, what a downer...
Hobby scale you said ? It doesn't have to be that complicated.
This is only tangentially related, but I have a love-hate relationship with Immich. Love for everything, but loads of hate for the bug-o-feature that disallows sharing of albums if the website is not hosted over HTTPS.
I can fiddle with the page content in Developer Tools to make it work, but frankly, WTF, especially since Immich expects people to set up a reverse proxy for HTTPS access on their own and does not help them in any way (like providing a pre-configured caddy in their compose.yaml, I don't know)
Honestly, this is a problem that has been solved by LLMs. Just point Claude Code to the server over SSH and say "We need Immich to be served over HTTPS".
You could theoretically upstream the fixed compose.yaml if you felt like it, but the project seemed unusually hostile towards the one fix I tried to upstream.
Here's the entirety of the Caddy config to reverse proxy something with managed tls certs. I feel like it's not that big a deal?
caddy passes on the source IP by default so it's even easier than that
the immich documentation covers this as well as nginx, apache and traefik https://docs.immich.app/administration/reverse-proxyI get more blind visitors than visitors with javascript disabled these days
All of those requirements seem weird. I selfhost a few software and the user count is one. My extent at sysadmin is to write a systemd service, configure the proxy (if it’s on a vps and I use subdomains), and maybe use ansible for scripting the provisioning and deploy.
It’s fine to publish recommendations how to tune configurations for specific usages. But I only need the app, not the various things that you may think I need. And I really want the app to only solve its essential problem. It should not take care of the IT part other than providing configuration samples.
And I do `docker compose up` now with a caddy server in front of docker, so I can just add a new site (sub domain or directory) that reverse proxies to that docker container's port. And the beauty is, that I have a skill for it for claude code, so I actually don't have to do any of that for my local self-hosted stuff.
I just call /publish-local-docker subdomain.mydomain.local
And if there isn't a docker compose, it will create one, and then run it, and update caddy server on its own.
How many of these problems PHP solves out of the box or makes irrelevant?
It solves static files... If you are extremely careful to set the executable properties of your PHP scripts.
No other.
It feels like there could be an entire longform piece on how "modern" web development norms arose because hobbyist/student web developers learned everything in an ad-hoc, local environment without access to, you know, a development server or a mature software stack.
All. People forget hiw performant the LAMP stack is.
There's an easier solution. Just don't have private data. Then you don't need SSL either.
Unless you also want to deal with security, backups, GDPR compliance, etc.
Perhaps the article's title should be, "The web has become a hostile, largely automated environment and naive hobbyists may find it difficult to coexist with professional scrapers, scammers, grifters, bots, hucksters, and cybercriminals."
This is quietly brilliant