Ask HN: How do you make sure your servers are up as a single founder?

My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn't able to handle, because I didn't expect it so never thought to test for it -- the issue was that I was trying to insert IDs into a uuid database field, but this new data had non-uuid IDs. Because the original IDs were always generated by me, I was able to guarantee that the data was correct, but this new data was not generated by me. Of course, sufficient defensive programming would have avoided this as this database error shouldn't have prevented other stuff from working, but my point is that mistakes get made (we're humans after all) and things do get overlooked.

The problem is, restarting my service doesn't prevent this external data from getting received again, so it would simply break again as soon as more is sent and the system would be in this endless reboot loop until a human fixes the root cause.

That's a problem that I worry about, no matter how hard I try to make my system auto-healing and resilient (I don't know of any way to fix it other than putting great care into programming defensively), but again, we're human, so something will always slip through eventually...

Some people are suggesting to out-source an on-call person. That seems to me like the only way around this particular case. (The other suggestions can still be used to reduce the amount of times this person gets paged, though)

Random things will go wrong that you can't predict. Boxes will die suddenly and without reason, even after months of working fine without changes, and always at the worst possible moment. Your system needs to be built to withstand that.

I'll take the opposite approach of everyone here and recommend against serverless, kubernetes, and Heroku/PAAS.

You are a solo founder. You should understand your infra from the ground up (note: not understand an API, or a config syntax, but how the underlying systems actually work in great detail). It needs to be simple conceptually for you to do that. If anything goes wrong, you need to be able to identify the cause and fix it quickly.

I've gone through this first-hand and know all the trade-offs. If you'd like, I'm happy to discuss architecture decisions on a call. Email is in my profile.

Otherwise you allow the problem to persist, pile up with other issues (also fixed by restarts, I assume) and implementing automated restarts in that manner reduces not only your uptime in uncontrollable manner, but also your code/infrastructure quality, increasing your tech debt beyond the point of recovery.

Friends don't let friends fixing things by restarting them ;)