Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced Redis with MySQL for inventory reservations and it scaled

shopify.engineering

Post reply on HN
Print view View original

171–180 of 281 posts

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#171

Post by resonious »

12387128 wrote: post #164
briga wrote: post #159

Earlier quoted context omitted.

What exactly makes it "obvious" that this is written by AI? I could totally believe that AI was used to generate parts of it, but I really don't get the sense that the whole thing was written that way. I've seen way worse examples on this site. As software engineers we are constantly told that we need to heavily use these tools for our daily work. So is it surprising that software engineers use the same tools as writ…

Because the post is just rambling without a clear intent or direction. Why do you need "oversell protection" if you have "transactions". Isn't the whole point of a "transaction" that it handles concurrency and disk failures?

I work with Shopify often and the whole oversell protection thing is like a huge joke.

Oversell happens because Shopify doesn't decrement inventory until payment is confirmed. And they apparently would rather die than change that invariant.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#172

Post by tonyhart7 »

ramon156 wrote: post #143

Why even have a blog when you can't be arsed to write the posts. This is so obviously LLM-written. I have a positive view of Shopify engineers, but this kind of made a dent in that confidence.

unfortunately its new normal now

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#173

Post by thunky »

energy123 wrote: post #165
briga wrote: post #159

Earlier quoted context omitted.

What exactly makes it "obvious" that this is written by AI? I could totally believe that AI was used to generate parts of it, but I really don't get the sense that the whole thing was written that way. I've seen way worse examples on this site. As software engineers we are constantly told that we need to heavily use these tools for our daily work. So is it surprising that software engineers use the same tools as writ…

Subheading and dot point spam, low density writing (the opposite of standard technical english), including useless detail (like enumerating stats on Shopify's scale), using contrastive parallelism, and other llm-isms. Even if it's not AI it's bad writing done by someone who has picked up AI's worst ticks. For example this subheading: > "The real bottleneck: connections, not CPU" That's two AI smells. AI likes to say…

The good news:

Your worst complaint has nothing to do with the overall content or accuracy of the post.

Just style bashing.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#174

Post by JamesSwift »

bijowo1676 wrote: post #91
idoubtit wrote: post #90

Earlier quoted context omitted.

> not the best design [...] So those engineers at Shopify worked hard for months on a more performant system, but they missed the obvious structure? They chose a complex denormalization for no good reason? It may be true, but I think it's presumptuous to belittle their work when we have only partial information. My guess is that they had good reasons to think that the more obvious ways would not scale. And from readi…

i also work in big tech and know that a lot of bullshit design creeps into system design and prod, because everyone is overworked, overstressed, wants to just get things done for the quarterly performance review as to not get shitcanned with severance re concurrency, it is not a big issue at all. stock exchanges deal with HFT traders and can easily deal with concurrency of orders. Same can be implemented with shopify…

Famously, stock is settled on a delay (and generally doesnt involve physical products that are not fungible). Im sure theres a lot to glean from how they handle concurrency but Im not sure they are solving the same problems.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#175

Post by misiek08 »

fauigerzigerk wrote: post #141
manbash wrote: post #8

> Instead of one row per item with a quantity column, we use one row per sellable unit. An item with 10 units has 10 rows. > But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination…

I agree, it does seem awfully complicated and there are quite a few pieces missing for this to be a complete solution. I'm a bit surprised about the scalability case against a simpler solution. This is not about Shopify's scale. We're talking about contention for a specific SKU of a specific seller at a specific warehouse location. How many shopping carts are competing for a single SKU at the payment stage at peak ho…

We observe 200+ (can’t say closer number) purchases per second for single SKU with good marketing and price.

The other thing that bothers me - why not real stable, but maybe „too old, medieval” solution with Redis as the main source of through - without any sync with SQL at all in terms of stock… it worked in my previous job with much higher traffic (1000s/s). Yup, we ended up with app-side sharding, but it was stupid-simple.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#176

Post by muragekibicho »

briga wrote: post #159
ramon156 wrote: post #143

Why even have a blog when you can't be arsed to write the posts. This is so obviously LLM-written. I have a positive view of Shopify engineers, but this kind of made a dent in that confidence.

What exactly makes it "obvious" that this is written by AI? I could totally believe that AI was used to generate parts of it, but I really don't get the sense that the whole thing was written that way. I've seen way worse examples on this site. As software engineers we are constantly told that we need to heavily use these tools for our daily work. So is it surprising that software engineers use the same tools as writ…

Taken verbatim "But the hardest lesson wasn't about database design. It was discovering that the real bottleneck wasn’t what we were observing and measuring. "

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#177

Post by giovannibonetti »

I wonder how we could handle that in a simpler way with durable workflows (e.g. Temporal, Restante, DBOS) – which are similar to Erlang processes but with persistent disk storage. This could avoid the need to maintain the 1000 row inventory.

Perhaps each shopping cart would have its own workflow, and the inventory item would have one as well. Then, whenever a customer put an item in their cart, their cart workflow would send a signal to the inventory item workflow and wait for the response. The inventory item workflow would maintain a ledger controlling to which cart each unit goes, and it could batch the writes to this table. This way, even if 100k customers try to purchase the same item in the same second, it should handle the load.

After the batch is written to the ledger, the inventory item workflow would reply signals to each cart workflow confirming that the reservation was completed. The end-to-end latency from the consumer point of view would be a fraction of a second, without needing the 1000-row hot-inventory heuristic.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#178

Post by asG123k »

thunky wrote: post #173
energy123 wrote: post #165

Earlier quoted context omitted.

Subheading and dot point spam, low density writing (the opposite of standard technical english), including useless detail (like enumerating stats on Shopify's scale), using contrastive parallelism, and other llm-isms. Even if it's not AI it's bad writing done by someone who has picked up AI's worst ticks. For example this subheading: > "The real bottleneck: connections, not CPU" That's two AI smells. AI likes to say…

The good news: Your worst complaint has nothing to do with the overall content or accuracy of the post. Just style bashing.

Just? With all the Claudisms it is hard to distill what really happened. My guess:

- Our oversell protection was a gross hack that broke ACID.

- MySQL has a new feature that allows us to remove parts of the gross hack.

- Question: To what extent does the hack still exist?

Instead you get garbage like "The answer is often in the plumbing, not the engine." and "Crucially, this wasn't about making reservations fast. It was about making them safe neighbors."

But this crap is what Lütke wants, so they deliver.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#179

Post by kandros »

giovannibonetti wrote: post #177

I wonder how we could handle that in a simpler way with durable workflows (e.g. Temporal, Restante, DBOS) – which are similar to Erlang processes but with persistent disk storage. This could avoid the need to maintain the 1000 row inventory. Perhaps each shopping cart would have its own workflow, and the inventory item would have one as well. Then, whenever a customer put an item in their cart, their cart workflow wo…

simpler

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#180

Post by thunky »

asG123k wrote: post #178
thunky wrote: post #173

Earlier quoted context omitted.

The good news: Your worst complaint has nothing to do with the overall content or accuracy of the post. Just style bashing.

Just? With all the Claudisms it is hard to distill what really happened. My guess: - Our oversell protection was a gross hack that broke ACID. - MySQL has a new feature that allows us to remove parts of the gross hack. - Question: To what extent does the hack still exist? Instead you get garbage like "The answer is often in the plumbing, not the engine." and "Crucially, this wasn't about making reservations fast. It…

Did you read TFA? Because all of your questions are answered clearly there.
Post reply on HN