31–40 of 132 posts
And then it needs to rebuild an index on top of the already above-average load.
This is one of the best incident reports I think I've ever read, detailed, honest and without recrimination. I'm reminded of something I read in a book a while back about aviation disasters "It's not the first problem that kills you, it's the second". In this post the described system for change management is at least as good as any I've seen in production and yet a series of small problems got out of hand quickly.
Totally agree. Came here to say pretty much the same thing. On a related note, highly recommend the Checklist manifesto (book) for devops.
There's a trend towards "hosted POS", where point of sale systems have to talk to the "cloud" to do anything, even handle cash. Until recently, most POS systems were running off a server in the manager's office, which communicated to servers and credit card systems elsewhere. A network outage didn't affect cash transactions. Often, the systems could even process a credit card transaction without external help, giving up real-time validation but still logging the transaction for later processing.
There's a single point of failure being designed into retail here.
[1] http://www.usatoday.com/story/tech/2015/04/24/starbucks-poin...
They clearly just need to add Devops. That'll fix everything.
This issue was caused by a failure in communication between team members. That communication is just as important as good engineering.
I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…
To me it seemed that it was a bug in the tooling that split the index change into two separate change requests. I'm sure a change request supports more than one piece of DDL, and it must have worked in the normal case, otherwise they would have run into this problem much earlier. So it was likely some weird corner case.
Now that I think about it, this could happen to us as well. We have peer review for each database change, but only at the source level (so, definition of the schema); the actual commands for schema changes are usually generated automatically. If some bug in that step existed that was only triggered in weird corner cases, we'd be screwed.
A backup of our main database takes about 7 hours to restore, and then we'd have to reply the binlogs from the time of the last snapshot up to the failed schema changes, so I guess we'd lose about a solid work day if things went down south. Yikes.
What is the point of having a human "database operator" who carries out simple tasks like deleting an index when it shows up in a work request log? Is this how most companies structure their dev teams? If you have a human there to perform tasks, then it would seem natural to allow them "human advantages" such as the ability to communicate with the person who created the request, or the ability to have some of their o…
I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…
I was thinking the same thing. If nothing else, I don't understand the role of "database operator" if they'll just blindly delete a critical index without thinking about it. Shouldn't that person have known better than anybody how critical the index was?
One is to keep it running, monitor disc space, response times etc. but otherwise leave the schema to the developers.
Or you can own the schema, discuss all the changes and migrations with the developers etc.
If it was the first kind of DB operation (which wouldn't surprise me, because that's what our $work has as well), it's not surprising that they trust the developers to provide sane DDL patches.
We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task turns these into alerts to database operators to run pre-defined tools that handle index operations.
In this situation, an application developer didn't add a new index description or remove an index description, but rather modified an existing index description. Our automated tooling erroneously handled this particular change and interpreted it not as a single intention but instead encoded it as two separate operations (an addition and a removal).
Developers describe indices directly in the relevant application/model code to ensure we always have the right indices available -- and in part to help avoid situations like this. In addition, the tooling for adding and removing indexes in production is restricted to a smaller set of people, both for security and to provide an additional layer of review (also to help prevent situations like this). Unfortunately, because of the bug above, the intent was not accurately communicated. The operator saw two operations, not obviously linked to each other, among several other alerts, and, well, the result followed.
There are some pretty obvious areas for tooling and process improvements here. We've been investigating them over the last few days. For non-urgent remediations, we have a custom of waiting at least a week after an incident before conducting a full postmortem and determining remediations. This gives us time to cool down after an incident and think clearly about our remediations for the long-term. We'll be having these in-depth discussions, and making decisions about the future of our tooling and processes, over the next week.
What's so striking about this is that entire retail chains can be shut down by a problem in some cloud server. Starbucks had a server outage in April which caused stores to close.[1] There's a trend towards "hosted POS", where point of sale systems have to talk to the "cloud" to do anything, even handle cash. Until recently, most POS systems were running off a server in the manager's office, which communicated to ser…