121–130 of 281 posts
"But the hardest lesson wasn't about database design. It was discovering that the real bottleneck wasn’t what we were observing and measuring."
It's honestly weird Claude converges on this language because it's incredibly wordy and hard to parse. One would think semantic density would win out in training.
Both sucks in French but at least chatgpt prose is readable while Claude is awful.
Is it really the right choice to drop Redis and go back to a disk based relational database just to wrap transactions into a single unit? Redis handles tens of thousands of concurrent connections in a single event loop, while MySQL uses one thread per connection. No matter how I look at it, that seems like a step backward. Of course, performance isn't everything. And if performance isn't a problem, having everything…
No persistence means the data gets lost if machine shuts down or process crashes. Furthermore, after restart you will need to regenerate the data which can take time. That's why Redis is a cache and not a database. You can fix the persistence issue (Redis can write WAL log, don't remember if it does fsync or not), but then Redis won't be able to handle those thousands of concurrent connections.
Redis (and other NoSQL storages) don't have some magic architecture that gives them advantages over SQL databases. They just cut corners on ACID guarantees and skip fsync. Once you start doing fsync, your transaction throughput will drop to SQL database level.
Redis also doesn't have transactions which means every app error damages the data. You will spend engineer hours investigating and fixing the problems. Transactions save so much time and worries.
Earlier quoted context omitted.
> not the best design [...] So those engineers at Shopify worked hard for months on a more performant system, but they missed the obvious structure? They chose a complex denormalization for no good reason? It may be true, but I think it's presumptuous to belittle their work when we have only partial information. My guess is that they had good reasons to think that the more obvious ways would not scale. And from readi…
You might not have noticed that essentially the entire blog post was AI written. There's even this bit where they discover a remarkable trick: > Each round trip to the database has a cost. For carts with multiple line items, we batch reservation queries using UNION ALL so we fetch all needed units in one round trip Insights like that really don't read like senior level output, and of course, it's LLM output. I'm not…
Is it really the right choice to drop Redis and go back to a disk based relational database just to wrap transactions into a single unit? Redis handles tens of thousands of concurrent connections in a single event loop, while MySQL uses one thread per connection. No matter how I look at it, that seems like a step backward. Of course, performance isn't everything. And if performance isn't a problem, having everything…
Redis doesn't have transactions and persistence. No persistence means the data gets lost if machine shuts down or process crashes. Furthermore, after restart you will need to regenerate the data which can take time. That's why Redis is a cache and not a database. You can fix the persistence issue (Redis can write WAL log, don't remember if it does fsync or not), but then Redis won't be able to handle those thousands…
But I think using only MySQL is unnecessarily expensive, just to get single transaction tracking for bug tracing. So the article's argument seems to be:
'Use only MySQL as a solution to the distributed transaction consistency problem between two different storage systems, Redis and MySQL!'
But I think using Redis is much more elegant. It's easier to scale. I'd even argue that something like Saga would be a better approach. Of course, we might just have different opinions. But in my experience, reducing layers always ends up making things more complicated in the long run.
p.s. We have different views, but I do think some of your points are valid, so I upvoted your comment
Earlier quoted context omitted.
I would call this one-row-per-contract-type, and this is the most general model for the problem (e.g. the model cannot be further broken down into finer level), thus, the most scalable model given storage is dirt cheap.
Storage might be cheap but search and RAM isn't
Is it really the right choice to drop Redis and go back to a disk based relational database just to wrap transactions into a single unit? Redis handles tens of thousands of concurrent connections in a single event loop, while MySQL uses one thread per connection. No matter how I look at it, that seems like a step backward. Of course, performance isn't everything. And if performance isn't a problem, having everything…
Redis doesn't have transactions and persistence. No persistence means the data gets lost if machine shuts down or process crashes. Furthermore, after restart you will need to regenerate the data which can take time. That's why Redis is a cache and not a database. You can fix the persistence issue (Redis can write WAL log, don't remember if it does fsync or not), but then Redis won't be able to handle those thousands…
Earlier quoted context omitted.
You might not have noticed that essentially the entire blog post was AI written. There's even this bit where they discover a remarkable trick: > Each round trip to the database has a cost. For carts with multiple line items, we batch reservation queries using UNION ALL so we fetch all needed units in one round trip Insights like that really don't read like senior level output, and of course, it's LLM output. I'm not…
There is now new type of comment in HN, if you disagree with article you attack the fact that ai was used in post editorial process. People here now dismiss anything that have em dash.
not the best design to have 1000 rows for each shop*SKU combination. If a candidate proposed this solution during Shopify's System Design interview, i doubt he would be vetted for Senior+ position. Instead of having 1000 rows per shop*SKU, why not just have one row per shopping cart*SKU? That way a single row would represent a single cart, and will hold info of multiple items of the same SKU. No need a cludge with 10…
I have never worked anywhere where describing how their system actually works would pass the company's own system design interview