41–50 of 132 posts
Earlier quoted context omitted.
I was thinking the same thing. If nothing else, I don't understand the role of "database operator" if they'll just blindly delete a critical index without thinking about it. Shouldn't that person have known better than anybody how critical the index was?
There is an open secret you may not be aware of: every startup is a shitshow on the inside.
Unless you're disabling a feature, don't push "quick code fixes". You'll pay for it later.
For schema/index changes I prefer migrations. These should be committed along side the code, tested, and promoted through environments. This largely prevents dependency issues because the migrations are ordered. Devops is like 80% dependency management. It's painful initially but you have to crack down on manual changes to production - all production changes should be defined in code, committed to git, tested, and fl…
Do people even test for perf issues in non-prod environments?
> At 00:08 UTC, our on-call engineer had been paged and had responded. At 00:10 UTC, we linked the API degradation to the removal of the index. I'm sure stripe has very good metrics of all their systems, nonetheless that's some rockstar level debugging skills.
I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…
This incident report reminded me of the 'Game Day Exercise' post from 2014:
https://stripe.com/blog/game-day-exercises-at-stripe
in which one robustness check that should be a continuous-integration kind of test, or at least a daily test of a normally working system, is such a big deal to them that they make a big 'Game Day' about it, and serious problems result from this one simple test.
After they have lots of paying customers, of course.
I know we are supposed to be positive and supportive on HN but this was a red flag that the entire department has no idea what an actual robust system looks like and were so far away from that, after having built a substantial amount of software, that expecting them to ever get there may be wishful thinking.
So I am completely unsurprised that they are having this kind of problem. The post-mortem reveals problems that could only occur in systems designed by people who do not think carefully about robustness ... which is consistent with the 2014 post. It kind of shocks me that anyone lets Stripe have anything to do with money.
What's so striking about this is that entire retail chains can be shut down by a problem in some cloud server. Starbucks had a server outage in April which caused stores to close.[1] There's a trend towards "hosted POS", where point of sale systems have to talk to the "cloud" to do anything, even handle cash. Until recently, most POS systems were running off a server in the manager's office, which communicated to ser…
The "2.0 generation" (or was that 5 years ago?) is forgoing a lot of redundancy and safeguards that existed in the previous systems
When did we assume a fast data link (and all the infrastructure behind it) would be available 24/7?
At least do cash transactions and note them in a notebook
Earlier quoted context omitted.
I was thinking the same thing. If nothing else, I don't understand the role of "database operator" if they'll just blindly delete a critical index without thinking about it. Shouldn't that person have known better than anybody how critical the index was?
There are different levels at which you can operate a database. One is to keep it running, monitor disc space, response times etc. but otherwise leave the schema to the developers. Or you can own the schema, discuss all the changes and migrations with the developers etc. If it was the first kind of DB operation (which wouldn't surprise me, because that's what our $work has as well), it's not surprising that they trus…
TBH, it seems odd to call the first one a "database operator" instead of an IT admin.
What's so striking about this is that entire retail chains can be shut down by a problem in some cloud server. Starbucks had a server outage in April which caused stores to close.[1] There's a trend towards "hosted POS", where point of sale systems have to talk to the "cloud" to do anything, even handle cash. Until recently, most POS systems were running off a server in the manager's office, which communicated to ser…
When Blue Bottle Coffee switches to Square there was a noticeable decline in the throughput at the cash register. It just takes the retail employee longer to do anything on an iPad. Pretty much everything can be done faster on a real cash register. There have also been the requisite outages, of course. Recently I was at Blue Bottle and the Square terminal wasn't opening the cash drawer. They were making change out of…
There are expensive retail consultants who clean up stores and improve sales by doing this.
Gap gets this right. Gap stores have big, clear counters, so you can bring up lots of merchandise and have a place to put it. This increases sales per customer. Gap is amazingly successful despite a rather blah product line.
> At 00:08 UTC, our on-call engineer had been paged and had responded. At 00:10 UTC, we linked the API degradation to the removal of the index. I'm sure stripe has very good metrics of all their systems, nonetheless that's some rockstar level debugging skills.
We had a utility I wrote at one of the places I worked where you ran it against the database and it showed you all of the query plans running. It was easy with 15 seconds to see some non-indexed query and what was executing it. We used to run it after new version deployments to see if we had query problems. Tooling is very important.