Backups not working when you need them is very very common. I've heard, experienced (for a client) and read it so very many times. In fact someone told me that pretty much all backups are useless unless you test them frequently (not just once!). There are lots of tools that do backups - not so many that check if the backup actually works. This might be a business opportunity for someone. It's not enough to check if t…
Best pattern I've seen to guarantee backup availability is for upgrades/deployments to occur on the hot failover first, swap the hot failover to primary, move primary to hot failover status, then upgrade what was previously the primary. This, combined with your general data backup (database dump to tape, etc.) gives you much more resilience should something go badly. However, it doesn't take the place of regularly testing your data backups, and only works if you have a full failover environment.