Live data from Hacker News

We replaced Redis with MySQL for inventory reservations and it scaled

shopify.engineering

261–270 of 279 posts

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#261
post #124

Earlier quoted context omitted.

Redis doesn't have transactions and persistence. No persistence means the data gets lost if machine shuts down or process crashes. Furthermore, after restart you will need to regenerate the data which can take time. That's why Redis is a cache and not a database. You can fix the persistence issue (Redis can write WAL log, don't remember if it does fsync or not), but then Redis won't be able to handle those thousands…

Does Redis become that slow when you enable both AOF and RDB? Sure, there's a write cost, but it doesn't lose its ability to maintain tens of thousands of connections. Redis supports AOF and lets you choose the fsync policy. But I think using only MySQL is unnecessarily expensive, just to get single transaction tracking for bug tracing. So the article's argument seems to be: 'Use only MySQL as a solution to the distr…

I became curious how fast fdatasync (a better version of fsync) is, and asked LLM to generate a microbenchmark for me. The benchamrk accepts records size, number of records, opens a file, and appends records one by one, making fdatasync after each one. On my SSD, the throughput is 120 fsyncs/second, or ~0.5 MB/s. So if I had Redis to flush the log after each operation, it wouldn't be able to serve those thousands of connections. That's why Redis is either used without AOF, or they flush the data once per second. And that's why SQL databases might look "slow".

Of course one could optimize this - for example, while fsync is being executed, we could accept the queries from other clients and execute them, and once previous flush finishes, flush multiple transactions at once. However, I am not sure if Redis can do this due to being single-thread.

And obviously maybe there are problems with drivers, or with my consumer-level SSD and maybe "professional" SSDs can do more flushes per second.

Writing the code took less than a minute. I often do microbenchmarks now because it is so easy.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#262

Earlier quoted context omitted.

It's likely because no one could be bothered to respond. There's plenty there, but it's not their job to fetch it for you. People on the Internet (especially here), seem to think that it's OK to make rude demands of complete strangers, and expect them to spend considerable time, meeting them. I'd rather some rando think they "won the Internet," than spend a bunch of my time, working for free, just to satisfy them. I…

> There's plenty there, but it's not their job to fetch it for you. It's been asked about 30 times now. If it existed, someone would have posted it. Maybe you.

No (see above)

One of the things that I learned, about the time I lost my baby teeth, is that making rude demands of people, instead of working with them, is likely to result in exactly the opposite.

Chances are good, that you learned this, as well, which might explain the rude demands. They are unlikely to be met, so the matter stays open.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#263

Earlier quoted context omitted.

> There's plenty there, but it's not their job to fetch it for you. It's been asked about 30 times now. If it existed, someone would have posted it. Maybe you.

No (see above) One of the things that I learned, about the time I lost my baby teeth, is that making rude demands of people, instead of working with them, is likely to result in exactly the opposite. Chances are good, that you learned this, as well, which might explain the rude demands. They are unlikely to be met, so the matter stays open.

If it can't be found, it can't be found *shrug

I mean, even the SOTA models couldn't pull up examples when asked, so it's no surprise that HN readers can't find any examples either.

They just don't exist...

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#264

Earlier quoted context omitted.

No (see above) One of the things that I learned, about the time I lost my baby teeth, is that making rude demands of people, instead of working with them, is likely to result in exactly the opposite. Chances are good, that you learned this, as well, which might explain the rude demands. They are unlikely to be met, so the matter stays open.

If it can't be found, it can't be found *shrug I mean, even the SOTA models couldn't pull up examples when asked, so it's no surprise that HN readers can't find any examples either. They just don't exist...

I apologize for my inferior communication skills. It appears as if I am unable to make the point clear. It has nothing at all to do with the matter at hand, and everything to do with communication style.

Have a great day!

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#265
post #67
post #12

Earlier quoted context omitted.

I found Shopify’s post very easy to read, and learned about some features of MySQL. On the other hand, I didn’t get any value from reading your comment. You seem to have a bunch of opinions about how things should be done, but haven’t given any details about how you came to these conclusions.

I've done multiple large scale implementations with shopify paired with many flavors of order managmeent systems as well as competing offerings in the space. The only one i haven't touched that i'd like to get my feet wet with is commerce tools. I really just disagreed with the assessment that redis is not good enough for the job for a reservation system. I use sql database all the time, I prefer them. But I'm seeing…

It’s unfortunate that the writing style wasn’t up to par, but I want to point out that I read the entire article and it doesn’t contain any criticisms of Redis. From the sounds of it, Redis worked perfectly and achieved what they wanted, which was beyond the capabilities of a simple MySQL implementation of the system.

In the end, they found a way to make it all work on MySQL, which solved some of the inherent complexity of using multiple databases side by side (but again this isn’t a criticism of Redis, the same problem would exist for any pair of databases).

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#266
post #73

Earlier quoted context omitted.

You should be concerned what they spot as the problem and what they propose as the solution https://www.ctvnews.ca/business/article/shopify-ceo-draws-cr...

The problem he was reacting to (local decisions dominated by people with locked-in benefits and low future exposure) is real. His idea there is silly but I think treating a couple of provocative X replies as some far right agenda is its own kind of overreach.

> The problem he was reacting to (local decisions dominated by people with locked-in benefits and low future exposure) is real

So like, idk, people over 60yrs old voting against climate action? Or maybe people making over $500k/year voting against a social safety net?

Those seem like far more consequential groups exerting influence than the very poor (who as a demographic don’t actually vote much anyway).

But anyway, whatever group you’re talking about disenfranchising, the fact that you’re talking about disenfranchising anybody makes you an enemy of democracy.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#267

Earlier quoted context omitted.

I work with Shopify often and the whole oversell protection thing is like a huge joke. Oversell happens because Shopify doesn't decrement inventory until payment is confirmed. And they apparently would rather die than change that invariant.

Would a retail business want the inventory to decrement just because users put items into carts that get abandoned? Seems like that would really mess things up more in the long run.

Yes. You decrement for a few minutes then release if there's no purchase in time. Airlines figured out how to do this a very long time ago.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#268

Earlier quoted context omitted.

You might not have noticed that essentially the entire blog post was AI written. There's even this bit where they discover a remarkable trick: > Each round trip to the database has a cost. For carts with multiple line items, we batch reservation queries using UNION ALL so we fetch all needed units in one round trip Insights like that really don't read like senior level output, and of course, it's LLM output. I'm not…

There is now new type of comment in HN, if you disagree with article you attack the fact that ai was used in post editorial process. People here now dismiss anything that have em dash.

You make a lot of assumptions here... that I disagree with something about the article (aside from with its quality) and post rationalized somehow, that it has anything to do with em-dashes, that they just used AI for "post editorial process". None of that is true.

What is there to disagree with? It's not making a statement or taking a stance, it's a technical writeup and I included a technical quote from the article which would be weird whether or not it was AI written.

It's not post edited by AI, it's written whole cloth by AI. No human cared enough to write it. Possibly if it was better quality you could make the case that's ok, but I suspect that you didn't read the article at all.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#269

Earlier quoted context omitted.

I actually don't think this article was LLM generated but these two sentences suck. I think they were moved from another part of the article without being modified. First, "the hardest lesson". What lesson? It is out of context. Nobody was talking about lessons before this. Second, "the bottleneck wasn't what we were measuring and observing". Of course the bottleneck itself wasn't that. They couldn't discover what th…

> but these two sentences suck. Sure. But that's not what's being discussed here. People are poor writers. People write, and they barely edit. But hard to parse? No, not at all.

It is hard to parse! It talks about some lesson that is out of context and the second sentence makes you do a double take because it's incorrect.

Re: We replaced Redis with MySQL for inventory reservations and it scaled

#270
post #124

Earlier quoted context omitted.

Does Redis become that slow when you enable both AOF and RDB? Sure, there's a write cost, but it doesn't lose its ability to maintain tens of thousands of connections. Redis supports AOF and lets you choose the fsync policy. But I think using only MySQL is unnecessarily expensive, just to get single transaction tracking for bug tracing. So the article's argument seems to be: 'Use only MySQL as a solution to the distr…

I became curious how fast fdatasync (a better version of fsync) is, and asked LLM to generate a microbenchmark for me. The benchamrk accepts records size, number of records, opens a file, and appends records one by one, making fdatasync after each one. On my SSD, the throughput is 120 fsyncs/second, or ~0.5 MB/s. So if I had Redis to flush the log after each operation, it wouldn't be able to serve those thousands of…

[deleted]
Post reply on HN