Self-Hosting Dozens of Web Applications and Services on a Single Server • Hacker News Board

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#271

Post by wiether » Sun, Jan 02, 2022, 10:27 AM UTC

amiantos wrote: post #216

Is there a reason people don’t self-host on a small home server? $85/mo is a lot of money if you aren’t making any money off what you’re hosting. If you mostly run CRUD APIs and sites, is there any downside to a low bandwidth home connection? Multiple family members already stream Plex content from my measly 20mbit upstream. Why not run a whole site or multiples?

I do both.

I have a server @home with Plex and few other services that are used mostly by myself and close relatives. Not something very public. It runs on a second-hand i7 NUC which is almost silent unless someone needs transcoding on content read from Plex. But it's "no warranty" services.

And I rent a $20 server at OVH on which I put more public stuff. Like blog for myself or others and for which I don't want to be linked _directly_. For instance a friend's blog was under attack because some people didn't shared her ideas. I was covered by OVH's anti-DDOS. If it had been on my home server, I probably would had some _troubles_. Same goes for the seedbox hosted on it. I don't have to set a VPN and put safety in place to make sure that not a single bit of data from the seedbox is shared without the VPN. I just run it exposing the public IP of the server and don't care. Worst case scenario is OVH taking down the server after (many) abuse reports.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#272

Post by wiether » Sun, Jan 02, 2022, 10:34 AM UTC

YPPH wrote: post #8

>I have all my HTTPS certificates issued + managed via letsencrypt. It's a terrific service; all I have to do is run snap run certonly --nginx every 3 months and everything is taken care of automatically. Hopefully through cron and not manual invokation! Certbot can safely be executed daily by cron, as it will only attempt renewal if it is required. Automating certificate renewal is an very important step in ensuring…

A few months ago I tried Nginx Proxy Manager[1] and never looked back.

It provides a nice looking UI to manage reverse proxy with Let's Encrypt certs, auto renewal and a few other nice features.

[1]: https://nginxproxymanager.com/

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#273

Post by dspillett » Sun, Jan 02, 2022, 11:53 AM UTC

justsomehnguy wrote: post #245

dspillett wrote: post #153

Earlier quoted context omitted.

I'm not sure this issue is significantly worse for SSDs compared to other drive types, except once they get really old and are close to EOL as defined by the amount written, though I'm happy to be proven wrong if you have some references where the theory has been tested. If you are really worried about that, perhaps artificially stress one of the drives for some days before building the array so it is more likely to…

Just add an event to calendar for $date+1y: "Replace one of SSD/NVMe drives in $server because I'm afraid of simultaneous failure (and probably still didn't configured backup for this server)"

> Just add an event to calendar for $date+1y

Unfortunately that won't wash if you are renting a server of collocating. Replacing a drive without it having already failed would most likely result in a charge for unnecessary hands-on maintenance time.

Though if you are expanding the storage anyway you could do it with an identically sized pair of drives, sync the existing array over to one of the new ones, drop the extra, then you have to unmatched drives for a new array. If using LVM you can join that to the existing VG or (less safe but better performing once you are done) you can try reshape the two arrays into a stripeset for RAID1+0. And hope that the new drives are not from the same batch as your existing ones and have just been sat in a store cupboard for the last year.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#274

Post by amaccuish » Sun, Jan 02, 2022, 1:32 PM UTC

fhackenberger wrote: post #270

lucb1e wrote: post #38

Earlier quoted context omitted.

It's an old thing that I should probably not mention for opsec reasons since, honestly, it probably can get pwned (I would be surprised if no intelligence agency has a 0-day for it), and I keep thinking I should switch to linux-based solutions, but it just never floats to the top of my todo list. It's also a fairly big operation since I'd have to coordinate the switchover with some people and I have no idea how to go…

I guess mailcow would suit your needs. Two years ago I replaced an old OpenXchange setup with mailcow and it was fairly easy to switch over. It also support IMAP copy from the admin UI.

What made you switch from OpenXchange?

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#275

Post by southerntofu » Sun, Jan 02, 2022, 2:17 PM UTC

remram wrote: post #265

southerntofu wrote: post #260

Earlier quoted context omitted.

> [edit: one app I host can only be run single-instance as it keeps some state in-process. It can outsource it to Redis, but that seems overkill if it's only needed during upgrades, 10s/week or so] Not exactly the same, but couldn't you serve a 425 Too Early with a meta refresh of 10s to achieve exactly the same goal? So change your nginx config to serve this "updating, your browser will refresh automatically" page,…

That might work for HTML pages, API routes would need a different configuration. Then every app consuming the API needs to be updated to support the 425 code gracefully. In addition, I have to manually do this switch in nginx before and after the update... I would rather a system that holds requests for 20s max if the backend refuses connections.

> I would rather a system that holds requests for 20s max if the backend refuses connections.

I think that's what nginx does by default, isn't it? Try again until "gateway timeout". The problem in this case is if you want graceful shutdown you need your backend to stop accepting new connections without stopping to process the existing ones, then update the app. If some client connections are long-lived, that's a hard problem and that's why in that case running a second instance makes sense if you can do it.

> Then every app consuming the API needs to be updated to support the 425 code gracefully.

That's indeed a problem, but your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply (or interpreting 5XX as such).

> I have to manually do this switch in nginx before and after the update...

This can be fully automated. Just use different config files in sites-enabled. Then you can mv or ln to enable one or the other and run nginx -s reload. The whole process would be a few lines of shell script.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#276

Post by SturgeonsLaw » Sun, Jan 02, 2022, 2:39 PM UTC

fivea wrote: post #262

SturgeonsLaw wrote: post #253

Earlier quoted context omitted.

If you have OP's level of control over the hardware (ie; you have a dedicated server on which you can access the hypervisor), then taking incremental backups of the entire VM is the best way to ensure you can hit the big undo button if anything goes wrong. The most important, and neglected, part of backup & restore, is the restore. If resources permit, this is where I like to use a battle tested solution instead of r…

> (...) taking incremental backups of the entire VM is the best way to ensure you can hit the big undo button if anything goes wrong. What if your server is already compromised? Doesn't that approach ensure your server remains compromised after you pressed the big undo button?

Potentially, depends when it happened, which point you restore to, and if the vulnerability that got them in in the first place is still present. If one of my servers got hacked I personally wouldn't risk it, and would just nuke it and rebuild. If everything's in Docker containers then the services can be torn down and spun up easily, and databases can be exported from a backup VM and imported into the new server.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#277

Post by slenk » Sun, Jan 02, 2022, 11:20 PM UTC

thrashh wrote: post #269

slenk wrote: post #268

Earlier quoted context omitted.

Have you ever looked at nginx proxy manager? I find it much easier to use than traefik since everything can be done through a UI. https://nginxproxymanager.com/

Yeah but once you figure out Traefik, it’s just 3 extra lines in your deployment files for every new service. And I inevitably have to redeploy again, and I hate doing the same boring thing twice, so it’s nice being able to bake complete orchestration into a repo. (And it’s also nice for being able to try things because your repo has a complete full snapshot of your setup that you can diff.)

The thing I like about nginxproxymanager is that it's easy to add non-docker hosts. There are some services that I route that I don't have in the same docker cluster as everything else. That requires static files changes for traefik itself somewhere.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#278

Post by rstat1 » Mon, Jan 03, 2022, 8:22 AM UTC

mholt wrote: post #250

rstat1 wrote: post #185

Earlier quoted context omitted.

To be fair, Caddy's docs aren't great either. Last time I tried to deploy it to give SSL to something that didn't have it, took me 2 hrs to figure out the docs enough to get it working.

What could make the docs better? What solved your problem?

Time and trial-and-error is what eventually solved it.

I was trying to configure the auto-HTTPS functionality to use DNS challenges, because my setup forbids any of the others, and this apparently requires plugins that are only supported by community efforts, so that's probably why I had such an issue.

The docs around configuring this were not immediately clear to me. Its like one part is over in this corner, the other over there. And the part about DNS provider plugins is off in a completely different direction. I definitely think things could be better organized. Also your one tutorial for getting HTTPS support only mentions the very basic cases. I think it'd be beneficial to see some more advanced tutorials using for instance some of the other LE challenge methods, something like this definitely would've saved me some time.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#279

Post by funcDropShadow » Mon, Jan 03, 2022, 1:59 PM UTC

forty wrote: post #257

FpUser wrote: post #129

Earlier quoted context omitted.

>"ability to scale" I write C++ business API servers. On the type of hardware I rent from Hetzner /OVH they're capable of processing thousands of requests per second. This would cover probably 90+ percent of real world businesses. >"an API for everything" Not sure what you mean here. API for what in particular? My servers talk to partner systems of real businesses. We consume their APIs and they consume ours. Integra…

Of course there are use cases where not using cloud facilities is cheaper, but unless your business is low margin the difference might still be irrelevant and having the managed services available easily can be handy if you ever need them. There are many use cases where hosting fee is not the right thing to optimize for.

The last time I compared hosting prices of AWS with self-hosting. AWS based hosting would have cost us every three month the cost of buying over the top speced server hardware. These numbers are not directly comparable, but they are a large difference to account for colocation and maintenance. And we never managed to get a similar performance of postgres on AWS compared to the self-hosted version we use now. But the main advantage is the simplification you get. It is certainly not easier to use you own hardware, but it is far simpler.

Re: Self-Hosting Dozens of Web Applications and Services on a Single Server

#280

Post by remram » Mon, Jan 03, 2022, 11:29 PM UTC

southerntofu wrote: post #275

remram wrote: post #265

Earlier quoted context omitted.

That might work for HTML pages, API routes would need a different configuration. Then every app consuming the API needs to be updated to support the 425 code gracefully. In addition, I have to manually do this switch in nginx before and after the update... I would rather a system that holds requests for 20s max if the backend refuses connections.

> I would rather a system that holds requests for 20s max if the backend refuses connections. I think that's what nginx does by default, isn't it? Try again until "gateway timeout". The problem in this case is if you want graceful shutdown you need your backend to stop accepting new connections without stopping to process the existing ones, then update the app. If some client connections are long-lived, that's a hard…

This is not my experience with nginx, I am seeing requests eventually time out. Then I am seeing 504 errors even though the backend is up, seemingly like it takes some time for nginx to notice.

This might very well be a configuration error on my part though.

> your client should probably support some forms of server "failure" modes, including a "try again soon" type of reply

Currently the "try again soon" is passed on directly to the user. For a lot of endpoints, the client is a web browser for which this is not possible (short of turning the whole thing into a PWA... just for the ~10s a week the app is restarting...)