Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We replaced Redis with MySQL for inventory reservations and it scaled

shopify.engineering

Post reply on HN
Print view View original

151–160 of 282 posts

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#151

Post by RobotToaster »

manbash wrote: post #8

> Instead of one row per item with a quantity column, we use one row per sellable unit. An item with 10 units has 10 rows. > But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination…

> an item with 50,000 units across 10 locations would mean 500,000 rows

I don't get it, Wouldn't that still only be 50,000 rows, just divided across locations?

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#152

Post by pythonRon »

Tobi Lütke also made some rather controversial statements lately, too, agreeing with a retired TD Bank CEO that more votes should be given to the rich. On the surface, this sounds awful, but what Eric Thor actually said was that the number of votes should be tied to the amount of income tax a person pays. Considering that (from what I've heard) billionaires pay no income taxes, I'd say it's not a bad idea. No tax: no vote.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#153

Post by fragmede »

RobotToaster wrote: post #151
manbash wrote: post #8

> Instead of one row per item with a quantity column, we use one row per sellable unit. An item with 10 units has 10 rows. > But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination…

> an item with 50,000 units across 10 locations would mean 500,000 rows I don't get it, Wouldn't that still only be 50,000 rows, just divided across locations?

they're saying they started with the dumb thing,

    item1_location1

    item1_location2

    ...

    item2_location1

    item2_location2
every (item, location) combo gets its own row, and then they moved to the smarter thing.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#154

Post by [deleted] »

neerajsi wrote: post #79
bijowo1676 wrote: post #68

Earlier quoted context omitted.

how does current design resolve concurrent actors fighting for the last item ? there is ultimately needs to be some global mechanism resolving this conflict. Currently it is an order in which db engine processes transactions by locking rows for a transaction, whoever got the first lock, wins the last remaining items. my design is the same, except it does not need this dance with moving rows between tables, locking th…

Clearly this is for high concurrency cases where there are many people racing to get all the available items. It's not clear that it's in shopifys or the sellers interest to let items get sequestered in people's shopping carts, which is a spot where there isn't a strong commitment to complete the purchase. At payment time, you can be more assured that the item will actually be purchased. Still I think their solution…

[deleted]

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#156

Post by fauigerzigerk »

kevincox wrote: post #149
fauigerzigerk wrote: post #141

Earlier quoted context omitted.

I agree, it does seem awfully complicated and there are quite a few pieces missing for this to be a complete solution. I'm a bit surprised about the scalability case against a simpler solution. This is not about Shopify's scale. We're talking about contention for a specific SKU of a specific seller at a specific warehouse location. How many shopping carts are competing for a single SKU at the payment stage at peak ho…

Flash sales are a huge scaling issue for Shopify. There are celebrities who want to sell thousands of items in a few minutes window at the end of an advertised countdown. Basically this is an incredibly rare case but a feature that they want to support.

Makes sense.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#157

Post by cowboylowrez »

ramon156 wrote: post #143

Why even have a blog when you can't be arsed to write the posts. This is so obviously LLM-written. I have a positive view of Shopify engineers, but this kind of made a dent in that confidence.

I just wanted to drop a note that I almost never go into an article or blog wondering if its AI or not. I am trying to be a more picky reader and try to actually do something occasionally but I do browse database articles and this is a good one. So I wonder, am I becoming insensitive about AI writing? Am I being hypnotized into taking whatever color pill thats had a color representing a surrender to the AI "hive mind" whatever that is?

So I'm sort of curious, did you not like the AI writing style or just rejecting AI in general? I myself am very conflicted, I think AI in its current form is the wrong tech at the wrong time yet I don't mind reading AI text and sort of mooch off of googles free tier.

Also I'm starting to notice lots of AI smearing, I read folks online describing someone elses contribution as "obviously AI" and I'm suspecting in some cases these could be false accusations.

I'm thinking about starting a blog, so when my AI gf writes posts, do I ask her to try to "not read like an AI"? Anybody try that? I'm gonna try that. Hehe for all you know I already did hehe

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#158

Post by grey-area »

paytonjjones wrote: post #22
zhivota wrote: post #6

"But the hardest lesson wasn't about database design. It was discovering that the real bottleneck wasn’t what we were observing and measuring."

It's honestly weird Claude converges on this language because it's incredibly wordy and hard to parse. One would think semantic density would win out in training.

Claude doesn’t think or have a goal like correctly summarise a topic, it just generates text based on the corpus and training. So not really weird that it is sometimes vague and sometimes incorrect.

Training could not address semantic density unless it was for the very specific pattern you need for this very specific article.

I guess the people generating the article don’t really care if is correct or easy to read, as long as it gets indexed in Google and linked by prominent sites like HN it has done its job (seo).

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#159

Post by briga »

ramon156 wrote: post #143

Why even have a blog when you can't be arsed to write the posts. This is so obviously LLM-written. I have a positive view of Shopify engineers, but this kind of made a dent in that confidence.

What exactly makes it "obvious" that this is written by AI? I could totally believe that AI was used to generate parts of it, but I really don't get the sense that the whole thing was written that way. I've seen way worse examples on this site.

As software engineers we are constantly told that we need to heavily use these tools for our daily work. So is it surprising that software engineers use the same tools as writing aids? Using AI does not mean no human effort was involved.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#160

Post by jdw64 »

codedokode wrote: post #150
jdw64 wrote: post #124

Earlier quoted context omitted.

Does Redis become that slow when you enable both AOF and RDB? Sure, there's a write cost, but it doesn't lose its ability to maintain tens of thousands of connections. Redis supports AOF and lets you choose the fsync policy. But I think using only MySQL is unnecessarily expensive, just to get single transaction tracking for bug tracing. So the article's argument seems to be: 'Use only MySQL as a solution to the distr…

The fsync policy equivalent to SQL database would be "fsync on each change before reporting successful update to the app". Redis (as many NoSQL databases) also doesn't have SQL and is a pain to view the data, you need to write extra tools when investigating the problems. RDB snapshots can cause multiple page faults due to use of fork() and CoW. > It's easier to scale The company in question manages online stores and…

It's interesting that our views differ. I think it's because of our different experiences. I believe MSA is the right approach. But this doesn't seem like a debate that can be resolved through discussion. It's something that needs to be implemented and tested.

Still, I respect your perspective and your experience. We clearly have different values, but I think you have a mature engineering mindset. Ultimately, I think only real measurements can settle this. Have a great day.

Post reply on HN