Scaling SQL with Redis
cramer.io
Scaling SQL with Redis
1–10 of 37 posts
Re: Scaling SQL with Redis
#2Redis really is a fundamental building block for designing distributed systems these days. I was kind of surprised, but all these examples exist independently in the Zapier codebase as well (all backed by Redis).
I've been meaning to open source our timeseries implementation for a while now, it is very similar to the linked article but uses a "{key}:YYYY:MM:DD:hh:mm:ss" pattern on hashes where you pick your stored granularity and TTL for each time unit. For example: store second granularity "{key}:YYYY:MM:DD:hh:mm": {0-60: count} for 8 hours, minute granularity "{key}:YYYY:MM:DD:hh": {0-60: count} for 24 hours, hour granularity "{key}:YYYY:MM:DD": {0-24: count} for 3 days, the rest forever. Very similar to https://github.com/jimeh/redistat or other implementations.
Fun!
Re: Scaling SQL with Redis
#3Redis really is a fundamental building block for designing distributed systems these days. I was kind of surprised, but all these examples exist independently in the Zapier codebase as well (all backed by Redis). I've been meaning to open source our timeseries implementation for a while now, it is very similar to the linked article but uses a "{key}:YYYY:MM:DD:hh:mm:ss" pattern on hashes where you pick your stored gr…
Re: Scaling SQL with Redis
#4Redis really is a fundamental building block for designing distributed systems these days. I was kind of surprised, but all these examples exist independently in the Zapier codebase as well (all backed by Redis). I've been meaning to open source our timeseries implementation for a while now, it is very similar to the linked article but uses a "{key}:YYYY:MM:DD:hh:mm:ss" pattern on hashes where you pick your stored gr…
And also https://github.com/antirez/redis-timeseries https://github.com/o/simmetrica https://www.npmjs.org/package/redis-timeseries
Oooo, simmetrica looks very nice! IIRC when I was writing our implementation there weren't any solid Python versions yet. Glad to see that changing!
More good implementation info here http://blog.apiaxle.com/post/storing-near-realtime-stats-in-....
Re: Scaling SQL with Redis
#5Redis is great... except for the fact it's (publicly) not ACID, so adding Redis in the mix and calling it "scaling SQL" is outright misleading, because it loses the very properties SQL exists to provide.
Redis will enter into conflicts (where in this article's example, those locks won't "lock" the thing you're locking), and it'll lose minutes of committed operations on unexpected stops.
Does that make Redis useless? Hell no. Can it help scale your app if carefully considered, with regards to its properties? Sure. Does it "scale SQL"? No.
Re: Scaling SQL with Redis
#6Redis is great... except for the fact it's (publicly) not ACID, so adding Redis in the mix and calling it "scaling SQL" is outright misleading, because it loses the very properties SQL exists to provide. Redis will enter into conflicts (where in this article's example, those locks won't "lock" the thing you're locking), and it'll lose minutes of committed operations on unexpected stops. Does that make Redis useless?…
[deleted]
Re: Scaling SQL with Redis
#7Redis is great... except for the fact it's (publicly) not ACID, so adding Redis in the mix and calling it "scaling SQL" is outright misleading, because it loses the very properties SQL exists to provide. Redis will enter into conflicts (where in this article's example, those locks won't "lock" the thing you're locking), and it'll lose minutes of committed operations on unexpected stops. Does that make Redis useless?…
Almost the entire post assumes you're not using it for durability. It's about making tradeoffs so the use of SQL can scale. If you want to be semantic, ACID will never be performant and scalable. The use of SQL on getsentry.com is already pushing Postgres to the limit's of what a locking/transactional database can do.
The locks are a minor bullet point in a much larger picture. Redis is never going to generate "conflicts" in a classical sense, but there are race conditions with the specific lock implementation. I definitely didn't suggest they were strong.
Re: Scaling SQL with Redis
#8Redis is great... except for the fact it's (publicly) not ACID, so adding Redis in the mix and calling it "scaling SQL" is outright misleading, because it loses the very properties SQL exists to provide. Redis will enter into conflicts (where in this article's example, those locks won't "lock" the thing you're locking), and it'll lose minutes of committed operations on unexpected stops. Does that make Redis useless?…
Almost the entire post assumes you're not using it for durability. It's about making tradeoffs so the use of SQL can scale. If you want to be semantic, ACID will never be performant and scalable. The use of SQL on getsentry.com is already pushing Postgres to the limit's of what a locking/transactional database can do. The locks are a minor bullet point in a much larger picture. Redis is never going to generate "confl…
I don't understand what's so hard to say the thing being scaled up is "the application domain model" and not "SQL". Not hard, is it?
A "scaling SQL" article that suggests adding Redis is like a "make more beer" article that suggests adding water.
There are performant algorithms for durable operations (as seen in frameworks like LMAX's Disruptor) which are simply not explored by Redis. The Disruptor is not canonical ACID, but it is durable.
They stumbled upon scalable durability because they had no other choice. As a trading platform, they were required to be durable by law, and required to scale by their clients.
A blanket all-or-nothing statement like "it will never scale" stops you before you even try to research the space of possible solutions.
Re: Scaling SQL with Redis
#9I love Redis so much, it became like a superglue where "just enough" performance is needed to resolve a bottleneck problem, but you don't have resources to rewrite a whole thing in something fast.
Re: Scaling SQL with Redis
#10Redis really is a fundamental building block for designing distributed systems these days. I was kind of surprised, but all these examples exist independently in the Zapier codebase as well (all backed by Redis). I've been meaning to open source our timeseries implementation for a while now, it is very similar to the linked article but uses a "{key}:YYYY:MM:DD:hh:mm:ss" pattern on hashes where you pick your stored gr…
Hey Bryan,
just curious, are you guys not already using statsd/graphite? I think that at least you used to, since you contributed to my small script[0] to automate the installation of graphite... So I'm curious if/why it wasn't good enough, or whether this has different requirements that graphite wasn't suitable for?