Live data from Hacker News

Stripe – Outage postmortem

support.stripe.com

121–130 of 132 posts

Re: Stripe – Outage postmortem

#121
post #49

Earlier quoted context omitted.

When Blue Bottle Coffee switches to Square there was a noticeable decline in the throughput at the cash register. It just takes the retail employee longer to do anything on an iPad. Pretty much everything can be done faster on a real cash register. There have also been the requisite outages, of course. Recently I was at Blue Bottle and the Square terminal wasn't opening the cash drawer. They were making change out of…

There's something that a lot of retailers don't get - never put an obstacle in front of the customer giving you their money. Don't let lines form at checkouts. Don't clog up the counter with impulse-purchase stuff. Don't put displays in the path of customers headed for checkout. Don't make customers jump through hoops with loyalty cards and data entry. Don't do anything that slows the checkout process. There are expe…

>Gap gets this right.

I did consulting for them and they hired aforementioned retail consultants for this job. I will admit I initially thought it was stupid, having been a wise retail manager at the age of 17 at one point in my life (and did a lot of level-1 tracking of sales and A/B testing, even back then!). But when these retail consultants showed me their data and how it was formatted... it was unbelievable. Most of what I experienced in retail were these nebulous and stupid ideas that were conjured from the ether from my direct managers and general managers and the like. While I'm sure not all the information I saw trickled down to employees, it was truly enlightening and gave me zero room to argue particulars with how they were studying and implementing retail flow, checkout, and return procedures.

Re: Stripe – Outage postmortem

#122

Earlier quoted context omitted.

I would encourage you to accept that their post mortem was released in good faith, and its purpose is to both technical knowledge sharing as well as PR. I know I personally value technical organizations that are honest and forthcoming when things go south.

> I know I personally value technical organizations that are honest and forthcoming when things go south. Why? How does the honesty (in this case openness really) change the quality? Genuine question: Would you rather have an org that's always reliable but private in their tech or one that has issues but open about them?

The second. Because "always reliable" isn't. So when something goes down and there is nothing being communicated, that's truly infuriating.

Re: Stripe – Outage postmortem

#123
post #3

Nicely done. Good job avoiding retrospectively blaming people and instead focusing on future system improvements. (For those wondering why this is important, Sidney Dekker's "Field Guide to Understanding Human Error" is a mindblowing book.)

Agreed. It looks like this was caused not because the application developer or DBA caused an error, but because the system didn't allow for ticket dependencies.

Re: Stripe – Outage postmortem

#124
post #70

Earlier quoted context omitted.

I suspect that it's more common than you'd guess. In one situation I was working for a supplier to a massive corporation (a household name). They had all sorts of red tape in the process. Every change required filling in forms and getting official sign off from several parties before getting the release code over. Anyway, we had a database user that was so restricted the we couldn't run the install process of a new p…

I guess its very common, but this is a financial institution. What you describe as a solution strikes me as an amazing way to destroy production data. Upgrading every user can basically lead to one hell of an amazing outside attack. I now only have to get one user / password to compromise your database. A certain amount of red tape is a needed thing to make sure you don't affect your customer's business.

True, I didn't stop to consider the financial aspect.

Was more just pointing out that there's often this separation between the people making the changes and the people making sure it's safe to release. But really, the safe to release step is often not going to catch things – in many cases because it's barely checked.

To be fair, I only upgraded my user to give full access to my db and I revoked the permissions once the install script had finished running. I'm not a complete monster :-)

Re: Stripe – Outage postmortem

#125
Would be cool to start the report with a tldr, containing just the essence of the incident, sth like

"dev needed to update index and at time .. submitted two tickets for new ibdex creation and old index removal. at time .. Op processed the removal ticket first which caused outage in service .. It was alerted at time .. and on-call op identified it at time .. He proceeded to ...

Just for people who want to know what's happened but dont care for details.

Re: Stripe – Outage postmortem

#126

Earlier quoted context omitted.

> I know I personally value technical organizations that are honest and forthcoming when things go south. Why? How does the honesty (in this case openness really) change the quality? Genuine question: Would you rather have an org that's always reliable but private in their tech or one that has issues but open about them?

The second. Because "always reliable" isn't. So when something goes down and there is nothing being communicated, that's truly infuriating.

This particular post-mortem by Stripe makes me trust them less as it's a fairly simple mistake that shouldn't have been made.

Plenty of companies also communicate the status and that something is happening but don't fully expound on all the internal details. Not sure why it's such a big difference if they did. It feels like fake PR trust to me.

Re: Stripe – Outage postmortem

#127

Earlier quoted context omitted.

In part though that's because the tooling to do it easily absolutely sucks , the impedance mismatch (overused but in context here) between the two parts of the system causes a lot of the underlying issues, better tooling is a large part of the solution I think but I've not seen anything that would help and the surface area of a modern RDBMS is so large without even getting into vendor specific stuff I'm not sure what…

That's certain a great point! If there was a way to automatically test much of this I bet even the newest of engineers could stop this. Doing that is tough, hmm...

I think the only way you could do it on top of a RDBMS is to use a strict subset of features that are common (something that many ORM's already do) which reduce the problem scope down to something manageable, the issue then would be that there would always be the temptation to use something outside that subset and forgo the easier testing, fast forward and you have the same issue.

It would be interesting to build a RDBMS that enforced that subset by simply not allowing those features to be used/abused with support for many of the modern features (JSONB etc) but that is way beyond my area of expertise.

Re: Stripe – Outage postmortem

#128

Earlier quoted context omitted.

The second. Because "always reliable" isn't. So when something goes down and there is nothing being communicated, that's truly infuriating.

This particular post-mortem by Stripe makes me trust them less as it's a fairly simple mistake that shouldn't have been made. Plenty of companies also communicate the status and that something is happening but don't fully expound on all the internal details. Not sure why it's such a big difference if they did. It feels like fake PR trust to me.

> This particular post-mortem by Stripe makes me trust them less as it's a fairly simple mistake that shouldn't have been made.

You are, of course, entitled to your opinion. I don't think its going to hurt their business at all.

Re: Stripe – Outage postmortem

#129

Earlier quoted context omitted.

MongoDB doesn't track this information, unfortunately.

It looks like the latest version does: https://jira.mongodb.org/browse/SERVER-2227 The problem with MongoDB is that teams think they can get away by just setting it and forgetting it. Real companies have DBAs that monitor it and understand it and make a living through it. They're just trying to automate it using fancy ui's. That's what you get for trying to automate your DBAs.

3.1.x is a development branch and not intended for production use. When they release 3.2, MongoDB will support it.

Re: Stripe – Outage postmortem

#130
post #3

Nicely done. Good job avoiding retrospectively blaming people and instead focusing on future system improvements. (For those wondering why this is important, Sidney Dekker's "Field Guide to Understanding Human Error" is a mindblowing book.)

Agreed. It looks like this was caused not because the application developer or DBA caused an error, but because the system didn't allow for ticket dependencies.

Definitely. But it would have been very easy to yell at the developer ("you should have known not to do it that way") or the DBA ("why are you doing tickets out of order? you know we have to do deletes last!").

Especially after a dramatic event, those are very easy reactions to have, and they can sound very sensible.

Post reply on HN