21–30 of 54 posts
What are the best practices redarding rollbacks when the database is affected. I would think a large amount of overhead would be required.
Rainforest QA had a blog post on strategies a while ago.
I'm less enthused about 'rollbacks' being considered 'normal'. They signify something didn't go quite right with your unit/integration/qa process. IMO there should be at least a 'mini-postmortem' to understand why it was missed even if it's in an intentional blind spot. (i.e you made an explicit decision it wasn't worth the engineering resources to get the testing fidelity needed to catch the issue earlier). It's alm…
It is amusing, because we are basically saying it would be awesome to have a coredump when the crash happens. Which... used to be standard behavior but was essentially lost in most modern development environments. (Not to mention the tooling lagged in two directions. One to help you with the coredumps, and the other to make usable core dumps.)
As a high traffic customer of Google, I've been this person far too many times. [...] if it breaks, real users get affected, so canarying should be the first step in your deployment process, as opposed to the last step in testing. It's a fine pattern and all, but not an excuse to throw stuff at prod and see what happens.
The point of the canary is it gives you one last real world test where you can limit the damage if anything goes wrong. Without it, you'll have to assume your tests are perfect, and I doubt that's ever achieved in practice in a real environment, except maybe in avionics and space control systems.
As a high traffic customer of Google, I've been this person far too many times. [...] if it breaks, real users get affected, so canarying should be the first step in your deployment process, as opposed to the last step in testing. It's a fine pattern and all, but not an excuse to throw stuff at prod and see what happens.
What's a high traffic customer of Google?
https://cloud.google.com/customers/
Spotify, SnapChat, eBay are the examples I usually give when asked that question.
I'm not sure what numbers I am allowed to provide for any of them, but there's some public information available that gives you a sense of the scale involved:
https://labs.spotify.com/2016/03/03/spotifys-event-delivery-...
https://www.itnews.com.au/news/do-not-fear-the-cloud-ebay-mi...
I'm less enthused about 'rollbacks' being considered 'normal'. They signify something didn't go quite right with your unit/integration/qa process. IMO there should be at least a 'mini-postmortem' to understand why it was missed even if it's in an intentional blind spot. (i.e you made an explicit decision it wasn't worth the engineering resources to get the testing fidelity needed to catch the issue earlier). It's alm…
I'm less enthused about 'rollbacks' being considered 'normal'. They signify something didn't go quite right with your unit/integration/qa process. IMO there should be at least a 'mini-postmortem' to understand why it was missed even if it's in an intentional blind spot. (i.e you made an explicit decision it wasn't worth the engineering resources to get the testing fidelity needed to catch the issue earlier). It's alm…
On the other hand, what you're talking about shouldn't be that hard to implement. Just hook your rollback system into your issue tracker to create a post-mortem issue whenever a rollback is necessary, and assign it to whoever initiated the deployment (or their manager). Easy.
On the third hand, you might end up finding that a lot of your post-mortems end up looking something like "we don't have reproduceable builds and we made a managerial decision not to invest in that now". And now you just created a ton of recurring paperwork for everyone with little benefit.
I'm less enthused about 'rollbacks' being considered 'normal'. They signify something didn't go quite right with your unit/integration/qa process. IMO there should be at least a 'mini-postmortem' to understand why it was missed even if it's in an intentional blind spot. (i.e you made an explicit decision it wasn't worth the engineering resources to get the testing fidelity needed to catch the issue earlier). It's alm…
You are a CTO sitting atop very expensive hardware and software. Would you start removing deployment and runtime safety guards (such as a consumer-facing staging environment) because you want to "discipline coders and devops"?
There are places where post-mortems can turn into blame games, but in my experience such things are counter-productive to actually solving problems. Luckily, there are plenty of engineering organizations that do not make this mistake! :)