Live data from Hacker News

We replaced Redis with MySQL for inventory reservations and it scaled

shopify.engineering

51–60 of 281 posts

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#51
post #8

> Instead of one row per item with a quantity column, we use one row per sellable unit. An item with 10 units has 10 rows. > But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination…

you should, their design is not the best. There is middle ground between "one row per SKU" and "1000 rows per SKU".

Its called one row per shopping cart*SKU combo.

if two people order 100 and 500 items of the same SKU, respectively, the table should have only two rows: for order1 and order2. Not 600 rows.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#52
post #11
post #8

> Instead of one row per item with a quantity column, we use one row per sellable unit. An item with 10 units has 10 rows. > But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination…

I guess it depends on how the replenishment process works. Unless you're ordering over 1000 of an item, I doubt it would be a problem.

replenishment is an unnecessary cludge that only exists due to poor design. an "algorithmical smell" if you wish

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#53
post #36
post #29

Earlier quoted context omitted.

My understanding is: your proposal is not very different from what Shopify is doing except they are tracking 'reserved units' (one per row) and you are proposing tracking 'orders' as the temporary state to then reconcile back with inventory quantities.

Yes, at a high level. It doesn't rely on skip locked, which is not cheap at DB level. DB has to still typically run query and keep going until it finds an unlocked item. Deducting and checking inventory counts are simpler ops inside the DB.

This seems like what triggers are for and how we do similar type things. Update trigger on order does select for update on the inventory and increases/decreases it as appropriate.

I don't think you really need that even. An indexed lookup is fast and you don't need to store a computed quantity generally.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#54

not the best design to have 1000 rows for each shop*SKU combination. If a candidate proposed this solution during Shopify's System Design interview, i doubt he would be vetted for Senior+ position. Instead of having 1000 rows per shop*SKU, why not just have one row per shopping cart*SKU? That way a single row would represent a single cart, and will hold info of multiple items of the same SKU. No need a cludge with 10…

> Instead of having 1000 rows per shopSKU, why not just have one row per shopping cartSKU?

At what point that row is inserted?

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#55

not the best design to have 1000 rows for each shop*SKU combination. If a candidate proposed this solution during Shopify's System Design interview, i doubt he would be vetted for Senior+ position. Instead of having 1000 rows per shop*SKU, why not just have one row per shopping cart*SKU? That way a single row would represent a single cart, and will hold info of multiple items of the same SKU. No need a cludge with 10…

> Instead of having 1000 rows per shop SKU, why not just have one row per shopping cart SKU? At what point that row is inserted?

per my reading of the article, the protection is only needed for a few seconds, while payment is being processed by the payment system.

so the row is inserted when Payment is initiated, and row is deleted when Payment succeeds

  What is oversell protection?
  Reserve: When payment starts, we mark items as reserved (a short hold, e.g. several minutes).
  Claim: When payment succeeds, we permanently deduct quantity from the inventory ledger (source of truth).

but that system could be easily improved to reserve item when user Adds item to a cart, to prevent scenario when user adds item to a cart, goes through checkout, and after initiating payment gets "soldout error":

  1. Let user add item to a cart by default (happy path)
  2. Initiate async check in the background for SKU and quantity
  2a. The check sums up rows for all SKUs and compares to Inventory table (very cheap check since its done to only active shopping carts)
  3. After few seconds the check comes back, and we let user know that item is soldout, before/the moment user goes to Checkout.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#56

Earlier quoted context omitted.

> Instead of having 1000 rows per shop SKU, why not just have one row per shopping cart SKU? At what point that row is inserted?

per my reading of the article, the protection is only needed for a few seconds, while payment is being processed by the payment system. so the row is inserted when Payment is initiated, and row is deleted when Payment succeeds What is oversell protection? Reserve: When payment starts, we mark items as reserved (a short hold, e.g. several minutes). Claim: When payment succeeds, we permanently deduct quantity from the…

Ok, but before inserting you must ensure that inventory is not depleted, which means you need to know the count and you need to lock the row. So you still have contention on that item. Them having a 1k buffer allows not to take a lock on a single row every time, and only do it when buffer is empty

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#58
post #50

Earlier quoted context omitted.

Don't get shy now. Which slurs?

Why are you trying to get someone to repeat slurs?

Because "slurs" is vague and covers a wide variety of utterances that run the gamut from unprofessional to unemployable and I had a tingle of Spidey sense that OP might have chosen the vague phrasing specifically to inflate the sins of the nameless Shopify director in the mind of the reader.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#59

Earlier quoted context omitted.

per my reading of the article, the protection is only needed for a few seconds, while payment is being processed by the payment system. so the row is inserted when Payment is initiated, and row is deleted when Payment succeeds What is oversell protection? Reserve: When payment starts, we mark items as reserved (a short hold, e.g. several minutes). Claim: When payment succeeds, we permanently deduct quantity from the…

Ok, but before inserting you must ensure that inventory is not depleted, which means you need to know the count and you need to lock the row. So you still have contention on that item. Them having a 1k buffer allows not to take a lock on a single row every time, and only do it when buffer is empty

there is no need to lock the row, since you a dealing with a shopping cart, not individual item piece. when you run aggregate functions, lock is no needed, it is actually better to run it with SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; for aggregation

the check for oversold items is extremely cheap:

  with current_order as (
    select $SKU1, $q2 as quantity
    union
    select $SKU2, $q2 as quantity
  ),
  with carts as (
    select sku, sum(quantity) as reserved
    from active_carts
    group by sku
  ),
  with warehouse as (
    select sku, available_units
    from inventory
    group by sku
  )
  select * from current_order
  inner join carts using (sku)
  inner join warehouse using (sku)
  where warehouse.available_units - carts.reserved 
assuming there are indexes on sku field in both, results in efficient index seek and agg over 2 tables

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#60
post #50

Earlier quoted context omitted.

Why are you trying to get someone to repeat slurs?

Because "slurs" is vague and covers a wide variety of utterances that run the gamut from unprofessional to unemployable and I had a tingle of Spidey sense that OP might have chosen the vague phrasing specifically to inflate the sins of the nameless Shopify director in the mind of the reader.

Which slurs are ok, and which aren't?
Post reply on HN