Most of the suggestions here is suggesting ways of restarting services when they go down, which is a good start, but that doesn't actually solve the issue I hit last night... My system integrates with an external system and what happened is this external system started sending me unexpected data, which my system wasn't able to handle, because I didn't expect it so never thought to test for it -- the issue was that I…
Glib answer: Don't work alone! There's two ways to think about this: 1 - Your product might actually be too complex for a single-person business. You could rotate being on call for situations like this. This means that you'd have to make sure that sales are big enough to support an additional parter or two. 2 - Perhaps you need to simplify your product? Think more critically about error handling? I don't know the det…
The difference with having a large team is less that all possible failure cases will get protected against (although more eyes and code review does help), but more that someone can always be available to fix it when something unexpected happens.
In my particular case, the majority of the system kept running fine. The part that failed was a streaming system which receives updates in realtime from an external system. The error actually was localised to one particular type of updates, but that type stopped working because I didn't protect defensively enough against errors in that one particular case (I do have my database queries protected against errors, but this one slipped through). This caused other systems to not get these updates, so things that relied on them stopped working. Its not that they crashed, they just never received the updates they were waiting for.
Of course the fix is to trap all exceptions, log/notify, ignore and continue, so that at least one piece of bad update doesn't affect other updates, but again, my main point was that we're human, so can't possibly protect against everything that might cause a non-recoverable (without human intervention) error.
> Finally, I have basic sanity checks for things like making sure a string is really a UUID
Yes, I did add this too after I hit this issue and its a good point: validate EVERYTHING even if you generate it and think you can assume it will be good.
> Don't work alone!
That's the real solution, but sometimes its not possible.
Thanks for your detailed response, though, its appreciated.