31–40 of 54 posts
How we do this at Google? Okay, tell me how to roll back the crap Android 6 upgrade back to 5 on this Samsung tablet I have here.
Anyone find a good way to do this with AWS Lambda / API Gateway?
This post might also be of interest: https://blog.jayway.com/2016/09/07/continuous-deployment-aws...
Note, I have never done this, this is just how is approach it.
Earlier quoted context omitted.
What's a high traffic customer of Google?
Host a high-traffic site on Google's infrastructure. Since you can see the version number of the platform, it's obvious when they're rolling out changes. This has cause many partial outages until the change was (I assume) automatically rolled back. It's a little hard to take this advice from Google after being the victim of so many bad rollouts. Because we use a lot of services, we are far more likely to have problem…
What are the best practices redarding rollbacks when the database is affected. I would think a large amount of overhead would be required.
At my job the DB is backwards compatible by one release. This means we roll back code, not data. It takes a little longer to do something like delete a column, but it's worth it. Rainforest QA had a blog post on strategies a while ago.
One could imagine a test rig which created a db with test users in it, then ran v1 of the software, then v2, then v1 again, and checked v1 didn't crash on the now mutated datastructures.
Most datastore mutations might only happen on certain user actions and inputs, so making sure you get full coverage would be rather tricky.
Earlier quoted context omitted.
They touched on this in their SRE post last week: https://cloudplatform.googleblog.com/2017/03/reliable-releas...
From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second. I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try aga…
Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.
Yes, a Canary lets you limit the damage if some bug sneaks past testing. We've done it for over 10 years, with staged rollouts and automated crash statistics and such. The draw back is that prod needs to be tolerant of multiple versions. Which is usually a fine practice in itself, anyway!
That's not a drawback so much as it's a fact of life. In any large-scale (read: distributed) system trying to provide a high degree of availability, rolling upgrades are the only way code goes out and individual components need to deal with interacting with newer/older dependencies. You can constrain the matrix by only allowing current version plus one back running in production, or forcing deployment orders, and so…
With dedicated hardware, you need twice as much hardware. With cloud, you only pay double for 10 minutes during the rollout, which usually is very cheap.
Earlier quoted context omitted.
What's a high traffic customer of Google?
Host a high-traffic site on Google's infrastructure. Since you can see the version number of the platform, it's obvious when they're rolling out changes. This has cause many partial outages until the change was (I assume) automatically rolled back. It's a little hard to take this advice from Google after being the victim of so many bad rollouts. Because we use a lot of services, we are far more likely to have problem…
It is common for a system to work fine before and after a rollout, but during a rollout clients experience errors.
One might imagine downloading a big file for example, which takes an hour. If you are downloading that from http-server-v1, which is being upgraded to http-server-v2, there will be a small grace period for clients of v1 to complete their operations. That grace period in many datacenters is often 30 seconds. That means if your operation is long running, you would see a failure. The error code is usually HTTP 503, for which your client logic should retry the request with an exponential backoff.
If your client doesn't retry/resume the request, now you see the service as down, when in fact the error you are seeing is by design. It will happen for every release, but also when servers come and go for maintanance, or for a bunch of other reasons.
Good libraries will handle retries for you, but some don't properly, and thats a bug.
I'm less enthused about 'rollbacks' being considered 'normal'. They signify something didn't go quite right with your unit/integration/qa process. IMO there should be at least a 'mini-postmortem' to understand why it was missed even if it's in an intentional blind spot. (i.e you made an explicit decision it wasn't worth the engineering resources to get the testing fidelity needed to catch the issue earlier). It's alm…
Staged rollouts are part of the QA strategy (whether this is something to be aspired is another question).
I'm less enthused about 'rollbacks' being considered 'normal'. They signify something didn't go quite right with your unit/integration/qa process. IMO there should be at least a 'mini-postmortem' to understand why it was missed even if it's in an intentional blind spot. (i.e you made an explicit decision it wasn't worth the engineering resources to get the testing fidelity needed to catch the issue earlier). It's alm…
> They signify something didn't go quite right with your unit/integration/qa process Staged rollouts are part of the QA strategy (whether this is something to be aspired is another question).
Earlier quoted context omitted.
From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second. I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try aga…
At our place, we rollback every few weeks just to test the system, even if nothing appears abnormal. Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.