Live data from Hacker News

Stripe – Outage postmortem

support.stripe.com

101–110 of 132 posts

Re: Stripe – Outage postmortem

#101
post #49

Earlier quoted context omitted.

When Blue Bottle Coffee switches to Square there was a noticeable decline in the throughput at the cash register. It just takes the retail employee longer to do anything on an iPad. Pretty much everything can be done faster on a real cash register. There have also been the requisite outages, of course. Recently I was at Blue Bottle and the Square terminal wasn't opening the cash drawer. They were making change out of…

There's something that a lot of retailers don't get - never put an obstacle in front of the customer giving you their money. Don't let lines form at checkouts. Don't clog up the counter with impulse-purchase stuff. Don't put displays in the path of customers headed for checkout. Don't make customers jump through hoops with loyalty cards and data entry. Don't do anything that slows the checkout process. There are expe…

Oh my goodness, 1000x yes. I am so sick of retailers putting crap in front of their checkout counters (Best Buy, Walmart, Barnes & Noble, every Indian run gas station, and thousands of others). Like running me through a rat maze of bookshelves, display cases, and pallets full of unopened stock is somehow going to make me happy about buying from there. Every time I see this in a retail store, it annoys me to death and seriously taints my view on the shopping experience. If I don't have enough counter room to put my purchases on or if I can't find the freaking entrance to your checkout corn maze, I'm going to leave my purchases on the ground.

Ironically, I think "open" stores (like an Apple store) exhibit the same behavior. Even though there's technically no labyrinth to navigate through, good luck finding an efficient way to purchase a product, especially if it's just a small one. You first signal an Apple checkout fairy to grace you with her checkout wand, so that someone else can find you and swipe your card. If your purchase is small, they hardly even give you time of day. So stupid.

And the analogy can be extended to online stores too. Any store with a "just add this too" in the checkout experience invokes the same negative reaction. Though I am a heavy Amazon customer, their checkout process is just barely tolerable in this way. And think about GoDaddy and how miserably awful their checkout experience has been (and continues to be) throughout the years.

Re: Stripe – Outage postmortem

#102
post #98

Earlier quoted context omitted.

That's not a tooling defect, that's a process and personal knowledge of the system defect. No DBA should remove an index without knowing what is replacing it or that code got deployed to make it obsolete. Checking if that index is currently being used would be a minimum. If a DBA cannot stop a deploy you have process problems.

The post mortem specifically noted that it looked unused, presumably because the code in master now claimed to depend on the new index.

I do not mean by looking at the source code. I mean by looking at the production database that you are about to remove an index from. This is a financial institution, and I am really struggling with this post mortem to understand how they do not have more ceremony around production changes. Checks and balances are important.

Re: Stripe – Outage postmortem

#103
post #70

So, and engineer can just submit a production change that an database operator will execute? What is the ceremony around this and does Stripe employ DBAs for production. What is the review process for a production change?

I suspect that it's more common than you'd guess. In one situation I was working for a supplier to a massive corporation (a household name). They had all sorts of red tape in the process. Every change required filling in forms and getting official sign off from several parties before getting the release code over. Anyway, we had a database user that was so restricted the we couldn't run the install process of a new p…

I guess its very common, but this is a financial institution.

What you describe as a solution strikes me as an amazing way to destroy production data. Upgrading every user can basically lead to one hell of an amazing outside attack. I now only have to get one user / password to compromise your database.

A certain amount of red tape is a needed thing to make sure you don't affect your customer's business.

Re: Stripe – Outage postmortem

#104

Earlier quoted context omitted.

We had a utility I wrote at one of the places I worked where you ran it against the database and it showed you all of the query plans running. It was easy with 15 seconds to see some non-indexed query and what was executing it. We used to run it after new version deployments to see if we had query problems. Tooling is very important.

As a stopgap solution, enabling the slow query log can also be very helpful.

Yep. A lot of databases have some form of system tables that can tell you a lot about a running system with just some queries. I know Sybase and MS SQL server have the tables to find all the current queries and what they are doing. Ingress had a neat graph (seeing FSM should illicit a panic in most DBAs) and a log monitor. Read up on what is part of the database and write some tools.

Re: Stripe – Outage postmortem

#105

Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…

have you considered integrating index statistics into these changes? To take an example from mysql, there is the INDEX_STATISTICS table in information_schema that contains the current number of rows read from the index. Checking this twice with a one minute interval before applying the index drop could have shown that the index was under heavy usage, and might require human intervention.

MongoDB doesn't track this information, unfortunately.

Re: Stripe – Outage postmortem

#106
post #49

Earlier quoted context omitted.

There's something that a lot of retailers don't get - never put an obstacle in front of the customer giving you their money. Don't let lines form at checkouts. Don't clog up the counter with impulse-purchase stuff. Don't put displays in the path of customers headed for checkout. Don't make customers jump through hoops with loyalty cards and data entry. Don't do anything that slows the checkout process. There are expe…

Oh my goodness, 1000x yes. I am so sick of retailers putting crap in front of their checkout counters (Best Buy, Walmart, Barnes & Noble, every Indian run gas station, and thousands of others). Like running me through a rat maze of bookshelves, display cases, and pallets full of unopened stock is somehow going to make me happy about buying from there. Every time I see this in a retail store, it annoys me to death and…

Sure, that's your reaction and it makes sense.

But for every person like you, there are about 50 others who do browse through all the stuff and perhaps add more to their shopping. Not everyone is in a hurry and capitalizing on small additions and combo deals and sales signups can be great business for the retailer.

Re: Stripe – Outage postmortem

#107
post #71

Earlier quoted context omitted.

No. That went out decades ago. The Syufy chain of theaters (later Century) used to try to create lines by not opening up enough ticket windows. Then came video rental. Then came half-empty theaters. Retailers no longer have the power to make customers wait. Consumers have too many other buying options.

>Retailers no longer have the power to make customers wait. Consumers have too many other buying options. And yet Apple had a line 12+ hours before opening to release their 9th generation iPhone -- despite also taking orders online. Hopdoddy in Austin has a line out the door for lunch and dinner; they even have a person dedicated to managing it, along with waitstaff to take drink orders I think the line is alive and…

I absolutely adore Hopdoddy's and waiting in line is considered part of the experience to be honest.

Re: Stripe – Outage postmortem

#108

Earlier quoted context omitted.

Shouldn't developers understand how a database change is going to impact an environment based on the code they've written?

Yes they very much should! But in my, admittedly anecdotal, experience only the best / most senior ever do. Almost every junior or mid developer I've worked with (and a small handful of senior folks) not only have no idea how changes like this would impact the larger environment but many won't even care to look into it.

In part though that's because the tooling to do it easily absolutely sucks, the impedance mismatch (overused but in context here) between the two parts of the system causes a lot of the underlying issues, better tooling is a large part of the solution I think but I've not seen anything that would help and the surface area of a modern RDBMS is so large without even getting into vendor specific stuff I'm not sure what that would even look like.

Re: Stripe – Outage postmortem

#109
post #42

"Quick code fixes" almost always make the problem worse and just cause stress and anxiety. It's always better to simply tackle the root cause and fix that. Unless you're disabling a feature, don't push "quick code fixes". You'll pay for it later.

Except this seems to be counter factual? And the quick fixes were just temporary until the index was finished building. Seems like a perfectly fine thing to do to restore service during an outage, as long as they're rolled back or reviewed later.

> Seems like a perfectly fine thing to do to restore service during an outage, as long as they're rolled back or reviewed later.

Rolling back later seems to forgotten in most cases (unless it's disabling a feature). Then you end up with this weird behavior and code path that few remember. "Oh yeah we did that when...." You usually only rediscover it after that quick hack is a problem. This is decades of experience here.

Re: Stripe – Outage postmortem

#110
post #49

Earlier quoted context omitted.

There's something that a lot of retailers don't get - never put an obstacle in front of the customer giving you their money. Don't let lines form at checkouts. Don't clog up the counter with impulse-purchase stuff. Don't put displays in the path of customers headed for checkout. Don't make customers jump through hoops with loyalty cards and data entry. Don't do anything that slows the checkout process. There are expe…

On the other hand, having a line out the door seems to sometimes be good marketing.

I had not thought about this before the parent posted, but there is a difference between a line on the way in and a line on the way out. The first is attractive and the second is obnoxious. One of the reasons I love British pubs and Japanese ticket-machine restaurants is that you pay up front and when you are ready to go home you just leave.

Japanese convenience stores also get this right most of the time. There is almost never a line up at the cash register. As soon as one begins to form, the employees will drop whatever they are doing and open up a new register. At some places, the register can actually handle more than one customer at a time, so if you have someone fumbling for change, they can start tallying up goods for the next person in line.

Possibly I'm not typical, but I often spend a lot of time browsing in stores. As soon as I have decided that I'm done, though, I get very irritated at anything that slows me down. I had never really put my finger on it until reading the OP's post.

Post reply on HN