What are the best practices redarding rollbacks when the database is affected. I would think a large amount of overhead would be required.
How release canaries can save your bacon
11–20 of 54 posts
Re: How release canaries can save your bacon
#12I wonder how the botched Google Drive release issue from a weeks ago worked under this scenario?
Re: How release canaries can save your bacon
#13What are the best practices redarding rollbacks when the database is affected. I would think a large amount of overhead would be required.
They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releas...
Re: How release canaries can save your bacon
#14Yes, a Canary lets you limit the damage if some bug sneaks past testing. We've done it for over 10 years, with staged rollouts and automated crash statistics and such. The draw back is that prod needs to be tolerant of multiple versions. Which is usually a fine practice in itself, anyway!
That's not a drawback so much as it's a fact of life. In any large-scale (read: distributed) system trying to provide a high degree of availability, rolling upgrades are the only way code goes out and individual components need to deal with interacting with newer/older dependencies. You can constrain the matrix by only allowing current version plus one back running in production, or forcing deployment orders, and so on, but in modern systems (read: ones where you can't just say "We're taking everything down for 3 hours on Sunday to upgrade.") you can't escape non-atomic upgrades.
Re: How release canaries can save your bacon
#15What are the best practices redarding rollbacks when the database is affected. I would think a large amount of overhead would be required.
They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releas...
Thanks. Basically they do a db change inbetween rollouts with no feature yet. Clever.
Re: How release canaries can save your bacon
#16What are the best practices redarding rollbacks when the database is affected. I would think a large amount of overhead would be required.
They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releas...
From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second.
I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try again.
Re: How release canaries can save your bacon
#17Interesting. I've heard of this practice as 'one box' or 'one pod'. And canary used to mean, 'tests that run continously against your production stack.' I wonder which is more prevalent.
Probably depends on your workplace, but at Google canary has meant a subset of production running at a newer version at least since '07.
Re: How release canaries can save your bacon
#18Interesting. I've heard of this practice as 'one box' or 'one pod'. And canary used to mean, 'tests that run continously against your production stack.' I wonder which is more prevalent.
I think your definition comes from Amazon/AWS... haven't heard it much from outside Amazon.
Re: How release canaries can save your bacon
#19As a high traffic customer of Google, I've been this person far too many times. [...] if it breaks, real users get affected, so canarying should be the first step in your deployment process, as opposed to the last step in testing. It's a fine pattern and all, but not an excuse to throw stuff at prod and see what happens.
What's a high traffic customer of Google?
Host a high-traffic site on Google's infrastructure. Since you can see the version number of the platform, it's obvious when they're rolling out changes. This has cause many partial outages until the change was (I assume) automatically rolled back.
It's a little hard to take this advice from Google after being the victim of so many bad rollouts. Because we use a lot of services, we are far more likely to have problems. We seem to always be the canary.
That's not fun.
Re: How release canaries can save your bacon
#20I'm less enthused about 'rollbacks' being considered 'normal'. They signify something didn't go quite right with your unit/integration/qa process. IMO there should be at least a 'mini-postmortem' to understand why it was missed even if it's in an intentional blind spot. (i.e you made an explicit decision it wasn't worth the engineering resources to get the testing fidelity needed to catch the issue earlier). It's almost always better to catch issues earlier, even if you have super neat tooling that makes it easy to rollback.