Earlier quoted context omitted.
I'm going to try to build a self-help tool for unlocking level instances. Briefly: you get round-robined to a (healthy, importantly) GM machine the first time you try to start a level and that association needs to be fairly sticky. Once you've started the level, a piece of that box's memory is all your own until the level ends. To prevent a single player from getting memory on all the boxes we "lock" a level to a par…
Perhaps it could be as simple as keeping a hash of levels tied to a box. When your monitoring alerts that your box is unhealthy, run a process to remove all locks associated with the dead box allowing the player to refresh and restart the session? Also, maybe a new error, is getting mixed in, response from /gm/levels {"ok":false,"error":"exit status 1"}
You could use container migration to move people off dying nodes, or to redistribute the cluster.
Ultimately though, it's probably a pretty lean startup and there is only so much experience and effort you can apply to each area.