61–70 of 132 posts
Sigh, yet another example of hot shot teams using MongoDB just because it's new and sexy. Existing, established tools such as Oracle and Postgres would have offered lots of ways of avoiding such a problem.
Sigh, yet another example of hot shot teams using MongoDB just because it's new and sexy. Existing, established tools such as Oracle and Postgres would have offered lots of ways of avoiding such a problem.
Earlier quoted context omitted.
When Blue Bottle Coffee switches to Square there was a noticeable decline in the throughput at the cash register. It just takes the retail employee longer to do anything on an iPad. Pretty much everything can be done faster on a real cash register. There have also been the requisite outages, of course. Recently I was at Blue Bottle and the Square terminal wasn't opening the cash drawer. They were making change out of…
There's something that a lot of retailers don't get - never put an obstacle in front of the customer giving you their money. Don't let lines form at checkouts. Don't clog up the counter with impulse-purchase stuff. Don't put displays in the path of customers headed for checkout. Don't make customers jump through hoops with loyalty cards and data entry. Don't do anything that slows the checkout process. There are expe…
For schema/index changes I prefer migrations. These should be committed along side the code, tested, and promoted through environments. This largely prevents dependency issues because the migrations are ordered. Devops is like 80% dependency management. It's painful initially but you have to crack down on manual changes to production - all production changes should be defined in code, committed to git, tested, and fl…
Do people even test for perf issues in non-prod environments?
In a continuous delivery environment like this, you generally want to test and validate any change you make. Do any DBs support the idea of turning off an index (but keep updating it). That way you could disable it, keep going for a while, and then delete it once you're sure it's unused?
Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…
Hi, I work in infrastructure at Stripe and I'm happy to provide more insight. Several threads here have commented on our tooling and processes around index changes. I can give a bit more detail about how that works. We have a library that allows us to describe expected schemas and expected indexes in application code. When application developers add or remove expected indexes in application code, an automated task tu…
I appreciate the post-mortem and, of course, we've all been there. I have to say, though, that the cause is a little surprising. That DDL needs to be executed sequentially is pretty basic and known by, I'm sure, everyone in the engineering and operations organizations at Stipe. It surprises me that an engineering group that is obviously so clever and competent would come up with a process that lost track of the requi…
It's pretty easy to monday morning quarterback other org's choices or actions here. Every person posting here will have something in their org break at some point that was "obvious" to the rest of us. EDIT: Someone, somewhere is going to have a bad day because they didn't know what you did. This is why sharing knowledge is so important. That's part of what HN exists for! Share what you know! Help improve open source…
In that case I don't need to basically congratulate Stripe on messing up and then posting PR piece on how they messed up, especially when it's such a trivial mistake (by that I mean it's not anything technically interesting for what went wrong). I guess I'll concede that what is technically interesting is not objective - many things I consider complicated other would consider basic, so I don't really have the right to serve as the arbiter of what is technically interesting.
What happened? We dropped an index. Why? Bad tooling. Fix? Patch code, add index. Future fix? Vague goals to stop this from happening.
Though I might be taking it too far, I don't see why I need to give props to someone for doing messing up something relatively basic and then fixing it - don't people complain enough about kids getting participation trophies?
Anyway, don't mean to call out any specific Stripe engineer, it's failed process at multiple levels (guy who drops index has no visibility into DB?).
Very interesting. In a continuous delivery environment like this, you generally want to test and validate any change you make. Do any DBs support the idea of turning off an index (but keep updating it). That way you could disable it, keep going for a while, and then delete it once you're sure it's unused?
So, and engineer can just submit a production change that an database operator will execute? What is the ceremony around this and does Stripe employ DBAs for production. What is the review process for a production change?
In one situation I was working for a supplier to a massive corporation (a household name). They had all sorts of red tape in the process. Every change required filling in forms and getting official sign off from several parties before getting the release code over.
Anyway, we had a database user that was so restricted the we couldn't run the install process of a new product. I knew they ran the releases as SA so the start of my release script upgraded all the permissions of our user to give full access to everything.