Ask HN: How do you make sure your servers are up as a single founder?

I am going to add that Lambda cold starts are pretty overblown unless you really do have an application that is super sensitive to latency (i.e. you are doing stock trading or something else pretty latency crazy). Most applications can use Lambda and pretty much never worry about cold starts. Consistent traffic keeps Lambda's warm and I've seen services at their busiest suffer less than 1% coldstarts.

I've run a one-person business on my own servers for about eight years.

Honestly, the answer is learning how to manage anxiety and stress, particularly doing potentially destructive things under pressure. I think the psychological aspects of this are much more difficult than the technical ones.

If it helps, people are generally very understanding if you explain that you are a solo founder, and take reasonable steps to fix issues in a timely way. Most customers assume every company is a faceless organization; their attitude is much more forgiving when they learn they're dealing with a fellow person.

You cannot be on call 24/7 forever. You will burn out. If you can't hire someone you trust to take over part of this burden, then you have to accept the risk of sometimes not being able to log in for N hours if there is an outage (because you're camping with your spouse, etc.)

For very high-stress situations (database crash, recovery from backup) working from a checklist that you have tested is very valuable.

Good luck to you, and I hope you found useful answers in this thread!

I'm not a solo founder, but I run a number of servers that are heavily used - all with different software with varying amounts of reliability. I also allow other people to deploy code without checking with me first, just to keep things fun.

I have a few pieces of advise:

1. Make sure your service can safely fail and be restarted. What I mean is, if somebody is POST'ing data or making database changes, make sure you handle this safely and attempt some recovery. Something not being fully processed is okay as long as you are able to handle it.

2. Self-monitoring. I run all my systems inside a simple bash loop that just restarts them and pop me an email (i.e. "X restarted at Y" and then "X is failing to start" if it continues).

3. External monitoring via a machine at home that rolls the server back to a previous binary (also on the server). It also pulls various logs from the server, as well as the binaries, so they can be analyzed. Okay, it has some reduced functionality, but it's stable and will keep things going until the problem is fixed.

4. Make sure your service fails inconveniently - i.e. returns a `{"status":"bad"}` string or something, or defaults to a "Under maintenance page, please come back soon". Your service going down is one thing, but becoming completely unresponsive is quite another.

One thing I can't prepare for (which happens more than you think) is the server itself crashing, which as you say, means I'm randomly logging into a VPS console and rebooting. I use a bunch of different VPS providers and every one of them has a slightly different console.

After reading another recent post... Tinder for Founders, anyone?

If you are using regular ECS with EC2 - as opposed to Fargate, it just provisions regular EC2 instance with an agent already installed. You can ssh/rdp into the instance and troubleshoot it.

But yeah I did have a doozy of an issue with ECS but it was completely my fault. I create a cross account policy for ECR but left out the account that actually contained the registry. Then my containers were in a private subnet without any access to the internet (by design they were behind a load balancer) but they couldn’t get to the ECR endpoint. I just had to either assign a public IP address or use a private link.

Support helped me with both.