Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced Redis with MySQL for inventory reservations and it scaled

shopify.engineering

Post reply on HN
Print view View original

221–230 of 280 posts

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#221

Post by mattmanser »

stasomatic wrote: post #213
energy123 wrote: post #165

Earlier quoted context omitted.

Subheading and dot point spam, low density writing (the opposite of standard technical english), including useless detail (like enumerating stats on Shopify's scale), using contrastive parallelism, and other llm-isms. Even if it's not AI it's bad writing done by someone who has picked up AI's worst ticks. For example this subheading: > "The real bottleneck: connections, not CPU" That's two AI smells. AI likes to say…

That's two AI smells. AI likes to say vacuous punchy statements like "the final takeaway" or "here's the rub". Then contrastive parallelism "connections, not CPU". Contrastive parallelism should scarcely exist in technical writing, regardless of whether it's AI generated. Aren't the LLMs trained on a massive corpus of human written texts? If that stands, then they are doing what they were asked, kind of? I am also no…

There's a post training.step where they get humans to interact with it.

As I understand it, there is (or was) a step where they ask people what's the 'better' response.

These linguistic forms sound good the first time you hear themz even if they are rare in real speech, so rapidly got trained in.

Now they distill off previous models, I imagine these weird linguistic forms are quite hard to get rid of.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#222

Post by zdragnar »

stasomatic wrote: post #213
energy123 wrote: post #165

Earlier quoted context omitted.

Subheading and dot point spam, low density writing (the opposite of standard technical english), including useless detail (like enumerating stats on Shopify's scale), using contrastive parallelism, and other llm-isms. Even if it's not AI it's bad writing done by someone who has picked up AI's worst ticks. For example this subheading: > "The real bottleneck: connections, not CPU" That's two AI smells. AI likes to say…

That's two AI smells. AI likes to say vacuous punchy statements like "the final takeaway" or "here's the rub". Then contrastive parallelism "connections, not CPU". Contrastive parallelism should scarcely exist in technical writing, regardless of whether it's AI generated. Aren't the LLMs trained on a massive corpus of human written texts? If that stands, then they are doing what they were asked, kind of? I am also no…

There are distinctive patterns of language use that can be powerful when used sparingly. LLMs trained on a massive copus of human communication, picked effective patterns, and overuses those patterns to the point that it feels both artificial and underwhelming.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#223

Post by codedokode »

bijowo1676 wrote: post #209
codedokode wrote: post #119

Earlier quoted context omitted.

The problem is that in this case you have to do splits/merges. And while there are products that are sold by 100 units at a time, I think in most cases people by 1-2 items so the hassle might be not worth it. Also you might not understand the original problem. Imagine if 100 customers want to buy product A. One thread starts a transaction, searches for amount of product A and UPDATE's it and goes searching for other…

ok, lets model situation of 100 customers and one last remaining item. Who will get the last item? in shopify's design, it is a user who was the first to lock the row and have successful payment. Sounds good, but how often does it happen ? It's a rare and extreme case and they model their entire system after the rare even, and incur the overhead of 1000 rows per SKU per shop for all combination of SKU and shop_id for…

On a large scale "rare" events happen every day. That is why people use locks and transactions or other measures.

In case with shopify, they want to decide whether the user may place order or not, at the moment when the user clicks "Pay" or some other button. If the user cannot place an order, they are shown the error, if they can, the items are reserved and the user is redirected to the payment page. So payment is processed only after successful reservation, and reservation is made only if the user wants to pay. The similar system works for buying train tickets online in my country, for example.

In you case, when user A clicks a button, following happens (as I understand):

1 the server increments the counter

2 the server calculates available amount as (amount_in_stock - amount reserved by carts with time 3 if the amount is large enough, the server updates the "time" field for user's cart thus reserving the item

Imagine that at step 2 the user A sees that there is one item left. However before user A does step 3, another user B might reserve the item (complete all 3 steps), and proceed to the payment. Then user A then completes step 3 and proceeds to the payment too. Now we end up with both user A and B paying for the last remaining item which doesn't solve the stated problem. Shopify's solution doesn't have such issues.

This is a classical TOCTTOU situation. There were exploits against Linux kernel based on similar issues.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#224

Post by DetroitThrow »

stasomatic wrote: post #213
energy123 wrote: post #165

Earlier quoted context omitted.

Subheading and dot point spam, low density writing (the opposite of standard technical english), including useless detail (like enumerating stats on Shopify's scale), using contrastive parallelism, and other llm-isms. Even if it's not AI it's bad writing done by someone who has picked up AI's worst ticks. For example this subheading: > "The real bottleneck: connections, not CPU" That's two AI smells. AI likes to say…

That's two AI smells. AI likes to say vacuous punchy statements like "the final takeaway" or "here's the rub". Then contrastive parallelism "connections, not CPU". Contrastive parallelism should scarcely exist in technical writing, regardless of whether it's AI generated. Aren't the LLMs trained on a massive corpus of human written texts? If that stands, then they are doing what they were asked, kind of? I am also no…

>Aren't the LLMs trained on a massive corpus of human written texts? If that stands, then they are doing what they were asked, kind of?

I think you might be interested in reading in training data generation, training, and post training papers/articles. I think you might be surprised at how much intervention there is on some of these levels.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#225

Post by hanikesn »

bijowo1676 wrote: post #105
hanikesn wrote: post #99

Earlier quoted context omitted.

Most payment methods in the world don't support separate authorization and capture.

i dont know about the world, by authorize.net and Stripe, which work globally and work with global credit cards, they do support separate authorize and separate capture, which seems to be part of PCI standard https://docs.stripe.com/payments/place-a-hold-on-a-payment-m... https://support.authorize.net/knowledgebase/Knowledgearticle...

You'll quickly realize PCI mainly applies to the credit card industry and not to something like Europe's psd2 and sepa instant.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#226

Post by Semaphor »

thunky wrote: post #180
asG123k wrote: post #178

Earlier quoted context omitted.

Just? With all the Claudisms it is hard to distill what really happened. My guess: - Our oversell protection was a gross hack that broke ACID. - MySQL has a new feature that allows us to remove parts of the gross hack. - Question: To what extent does the hack still exist? Instead you get garbage like "The answer is often in the plumbing, not the engine." and "Crucially, this wasn't about making reservations fast. It…

Did you read TFA? Because all of your questions are answered clearly there.

Maybe they should show us their prompt so we don't have to read all the bullshitting to get to those answers

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#227

Post by dessimus »

resonious wrote: post #171
12387128 wrote: post #164

Earlier quoted context omitted.

Because the post is just rambling without a clear intent or direction. Why do you need "oversell protection" if you have "transactions". Isn't the whole point of a "transaction" that it handles concurrency and disk failures?

I work with Shopify often and the whole oversell protection thing is like a huge joke. Oversell happens because Shopify doesn't decrement inventory until payment is confirmed. And they apparently would rather die than change that invariant.

Would a retail business want the inventory to decrement just because users put items into carts that get abandoned? Seems like that would really mess things up more in the long run.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#228

Post by drstewart »

energy123 wrote: post #165
briga wrote: post #159

Earlier quoted context omitted.

What exactly makes it "obvious" that this is written by AI? I could totally believe that AI was used to generate parts of it, but I really don't get the sense that the whole thing was written that way. I've seen way worse examples on this site. As software engineers we are constantly told that we need to heavily use these tools for our daily work. So is it surprising that software engineers use the same tools as writ…

Subheading and dot point spam, low density writing (the opposite of standard technical english), including useless detail (like enumerating stats on Shopify's scale), using contrastive parallelism, and other llm-isms. Even if it's not AI it's bad writing done by someone who has picked up AI's worst ticks. For example this subheading: > "The real bottleneck: connections, not CPU" That's two AI smells. AI likes to say…

>if it's not AI it's bad writing

Is this post AI generated?

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#230

Post by bijowo1676 »

codedokode wrote: post #223
bijowo1676 wrote: post #209

Earlier quoted context omitted.

ok, lets model situation of 100 customers and one last remaining item. Who will get the last item? in shopify's design, it is a user who was the first to lock the row and have successful payment. Sounds good, but how often does it happen ? It's a rare and extreme case and they model their entire system after the rare even, and incur the overhead of 1000 rows per SKU per shop for all combination of SKU and shop_id for…

On a large scale "rare" events happen every day. That is why people use locks and transactions or other measures. In case with shopify, they want to decide whether the user may place order or not, at the moment when the user clicks "Pay" or some other button. If the user cannot place an order, they are shown the error, if they can, the items are reserved and the user is redirected to the payment page. So payment is p…

shopify is wrapping their entire dance with locking and moving rows inside a transaction. if you wrap step 1-3 inside transaction you will get same atomicity guarantee

but again, my idea was:

  1) do not use throwaway placeholder rows to imitate a single item
  2) do not rely on db engine to decide which transaction gets committed first (which customer gets the last item)
  3) model queue explicitly by introducing counter field that sorts and prioritizes customers' orders and decides which order gets fulfilled and which customers gets the last item
Post reply on HN