"Outrage postmortem"
Stripe – Outage postmortem
111–120 of 132 posts
Re: Stripe – Outage postmortem
#112Earlier quoted context omitted.
Generally you want your database migrations described in a straightforward manner for development; the migrations will contain a straightforward change from old to new (and back). With a live (busy) production database, it is often necessary to handle things differently to maintain up-time. As a simple example, to make an atomic change to a write-only table, you could create a copy of the table, alter the copy as nec…
Development databases normally have a small amount of data, so migrations should execute instantly or nearly so no matter how complex they are.
True, but I don't think it negates anything I wrote. You don't keep development migrations simple so they'll run quickly; you keep them simple so they're easy to create and understand. Writing migrations (whether automated or manual) for production is a separate task and even a separate skill from designing the database structure itself, so there's no reason why the two need to be (or should be) combined.
Re: Stripe – Outage postmortem
#113Earlier quoted context omitted.
Why not just use simple version controlled database migrations, and testing them in a test environment?
Generally you want your database migrations described in a straightforward manner for development; the migrations will contain a straightforward change from old to new (and back). With a live (busy) production database, it is often necessary to handle things differently to maintain up-time. As a simple example, to make an atomic change to a write-only table, you could create a copy of the table, alter the copy as nec…
Meant to write 'read-only' in the example there. Those steps wouldn't work well for a table that's being written to, since it could change in the process. Anyway, it was just an example.
Re: Stripe – Outage postmortem
#114Earlier quoted context omitted.
Yes they very much should! But in my, admittedly anecdotal, experience only the best / most senior ever do. Almost every junior or mid developer I've worked with (and a small handful of senior folks) not only have no idea how changes like this would impact the larger environment but many won't even care to look into it.
In part though that's because the tooling to do it easily absolutely sucks , the impedance mismatch (overused but in context here) between the two parts of the system causes a lot of the underlying issues, better tooling is a large part of the solution I think but I've not seen anything that would help and the surface area of a modern RDBMS is so large without even getting into vendor specific stuff I'm not sure what…
That's certain a great point! If there was a way to automatically test much of this I bet even the newest of engineers could stop this. Doing that is tough, hmm...
Re: Stripe – Outage postmortem
#115Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…
(Tedious disclaimer: my opinion, not speaking for my employer, etc)
I'm an SRE at Google, where postmortems are habitual. The thing that jumped out at me here is that a production change was instantaneously pushed globally, instead of being canaried on a fraction of the serving capacity so that problems could be detected. That seems like your big problem here.
(Of course, without knowing how your data storage works, it's difficult to tell how hard it is to fix that.)
Re: Stripe – Outage postmortem
#116Earlier quoted context omitted.
have you considered integrating index statistics into these changes? To take an example from mysql, there is the INDEX_STATISTICS table in information_schema that contains the current number of rows read from the index. Checking this twice with a one minute interval before applying the index drop could have shown that the index was under heavy usage, and might require human intervention.
MongoDB doesn't track this information, unfortunately.
It looks like the latest version does: https://jira.mongodb.org/browse/SERVER-2227
The problem with MongoDB is that teams think they can get away by just setting it and forgetting it. Real companies have DBAs that monitor it and understand it and make a living through it. They're just trying to automate it using fancy ui's. That's what you get for trying to automate your DBAs.
Re: Stripe – Outage postmortem
#117Earlier quoted context omitted.
Oh my goodness, 1000x yes. I am so sick of retailers putting crap in front of their checkout counters (Best Buy, Walmart, Barnes & Noble, every Indian run gas station, and thousands of others). Like running me through a rat maze of bookshelves, display cases, and pallets full of unopened stock is somehow going to make me happy about buying from there. Every time I see this in a retail store, it annoys me to death and…
Sure, that's your reaction and it makes sense. But for every person like you, there are about 50 others who do browse through all the stuff and perhaps add more to their shopping. Not everyone is in a hurry and capitalizing on small additions and combo deals and sales signups can be great business for the retailer.
[deleted]
Re: Stripe – Outage postmortem
#118Great postmortem report. I am very surprised to see that the DB ops simply deleted the index without even considering the consequences.
More importantly, why isn't the 2 changes part of a single migration file?
For e.g. in Django, south migrations can have both remove, create indexes in one file and executed together.
Also, like some of the guys mentioned, why is the update performed to your entire global cluster. Shouldn't it be incremental like for e.g. one availability zone at a time?
Re: Stripe – Outage postmortem
#119Earlier quoted context omitted.
I get that but I'm wondering who is this postmortem written for? For other engineers? Not entirely - it's seems to be written partly as PR piece. In that case I don't need to basically congratulate Stripe on messing up and then posting PR piece on how they messed up, especially when it's such a trivial mistake (by that I mean it's not anything technically interesting for what went wrong). I guess I'll concede that wh…
I would encourage you to accept that their post mortem was released in good faith, and its purpose is to both technical knowledge sharing as well as PR. I know I personally value technical organizations that are honest and forthcoming when things go south.
> I know I personally value technical organizations that are honest and forthcoming when things go south.
Why? How does the honesty (in this case openness really) change the quality?
Genuine question: Would you rather have an org that's always reliable but private in their tech or one that has issues but open about them?
Re: Stripe – Outage postmortem
#120Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…
(Tedious disclaimer: my opinion, not speaking for my employer, etc) I'm an SRE at Google, where postmortems are habitual. The thing that jumped out at me here is that a production change was instantaneously pushed globally, instead of being canaried on a fraction of the serving capacity so that problems could be detected. That seems like your big problem here. (Of course, without knowing how your data storage works,…
Yup.
This is one of our few remaining unsharded databases (legacy problems...), so we can't easily canary a fraction of serving capacity. However, one clear remediation we can implement easily is to have our tooling change a replica first, failover to it as primary, and, if problems are detected, quickly fail back to the healthy former primary.
Lesson learned. We'll be doing a review of all of our database tooling to make sure changes are always canaried or easily reversible.