Postmortem of Heroku's June 23 Downtime
status.heroku.com
Postmortem of Heroku's June 23 Downtime
1–10 of 23 posts
Re: Postmortem of Heroku's June 23 Downtime
#2This paragraph reads like a response to the criticism they received a few days ago for scheduling maintenance at 2pm PST:
On June 23rd we performed a credential roll on these Redis servers in our US cloud during a two hour scheduled maintenance window. Because we operate a service used globally, there is a less-than 10% difference in usage between so-called "peak hours" and “non-peak” hours. We scheduled maintenance for this time because it was not a peak time, but moreso because this period has high coverage from relevant engineering teams, should issues arise. By performing maintenance during this period, we were able to react more quickly and muster those teams within seconds.
Re: Postmortem of Heroku's June 23 Downtime
#3This paragraph reads like a response to the criticism they received a few days ago for scheduling maintenance at 2pm PST: On June 23rd we performed a credential roll on these Redis servers in our US cloud during a two hour scheduled maintenance window. Because we operate a service used globally, there is a less-than 10% difference in usage between so-called "peak hours" and “non-peak” hours. We scheduled maintenance…
What's the best way to muster an engineering team?
Re: Postmortem of Heroku's June 23 Downtime
#4This paragraph reads like a response to the criticism they received a few days ago for scheduling maintenance at 2pm PST: On June 23rd we performed a credential roll on these Redis servers in our US cloud during a two hour scheduled maintenance window. Because we operate a service used globally, there is a less-than 10% difference in usage between so-called "peak hours" and “non-peak” hours. We scheduled maintenance…
What's the best way to muster an engineering team?
Bugle reveille.
Re: Postmortem of Heroku's June 23 Downtime
#5This paragraph reads like a response to the criticism they received a few days ago for scheduling maintenance at 2pm PST: On June 23rd we performed a credential roll on these Redis servers in our US cloud during a two hour scheduled maintenance window. Because we operate a service used globally, there is a less-than 10% difference in usage between so-called "peak hours" and “non-peak” hours. We scheduled maintenance…
What's the best way to muster an engineering team?
The smell of pizza?
Re: Postmortem of Heroku's June 23 Downtime
#6This incident makes me think that services like Redis should support running with two sets of credentials at once in order to facilitate credential rolling. As it currently stands, rolling credentials is a rather big deal with a chance of things going wrong in the process.
Aside: the text on that page is extremely difficult to read because of poor contrast (#8584B2 on #282936) and might be impossible for people with vision impairment. If anyone from Heroku is reading, you should change the color scheme to be compliant with the W3C Web Content Accessibility Guidelines. See: http://www.snook.ca/technical/colour_contrast/colour.html
Re: Postmortem of Heroku's June 23 Downtime
#7Because I so rarely feel compelled to say this: this is a really great post-mortem. It's technical, it's not loaded down with sales-speak, and it's straightforward. I really hope post-mortems like this become more of a trend.
Re: Postmortem of Heroku's June 23 Downtime
#8[deleted]
Re: Postmortem of Heroku's June 23 Downtime
#9Seems like first trying this maintenance procedure in a staging environment would have caught the problem.
Re: Postmortem of Heroku's June 23 Downtime
#10This incident makes me think that services like Redis should support running with two sets of credentials at once in order to facilitate credential rolling. As it currently stands, rolling credentials is a rather big deal with a chance of things going wrong in the process. Aside: the text on that page is extremely difficult to read because of poor contrast (#8584B2 on #282936) and might be impossible for people with…
I thought the exact same thing, but then I realized that we don’t need Redis to actually support a choice of two passwords for a single account. Rather, clients can be configured with a list of credentials to try. When rolling credentials over, simply add the new ones to the clients’ lists, update the service, then remove the old ones from the clients. Then you can wait hours or days between steps for safety, and there is no time when system reliability is degraded by a service instance being inaccessible.