Live data from Hacker News

Redis 8.8: New array data structure, rate limiter, performance improvements

redis.io

41–50 of 118 posts

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#41
post #7

Where did everyone end up on the Redis/Valkey split? Is there still a reason to use Redis after the license kerfuffle?

We're a self hosted shop, we went with Valkey. Valkey also has support for RDMA, which we already is running in our infrastructure.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#42
post #40
post #30

Earlier quoted context omitted.

It’s not. Imagine a web app that stores your user information in a session store, mapped by your cookie-provided session ID. Your web app searches redis 1 for the session id, but since that key is on redis 2, the lookup fails and the application thinks there is no such session, and rejects the request. Now you could solve this specific case by sharding by prefix, or by querying all instances, but then you still do no…

But that is his point. If you cannot find the session id in redis, you login again. If your Redis server crash, you start a new one and everyone just login again. No data is lost.

Sure the data is lost. A session commonly holds arbitrary state, and even if it’s just the login information. This is ridiculous.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#44
post #7

Where did everyone end up on the Redis/Valkey split? Is there still a reason to use Redis after the license kerfuffle?

We switched to Valkey two years ago. I haven't really looked back. I think both projects have done a lot of nice stuff since the split but it's not really impacting anything I use. The feature set was fine five years ago and I don't think we're using anything in Valkey that wouldn't work in Redis. There are probably a lot of projects that never switched over because they had no real need.

But most of the cloud providers now offer Valkey because of the license changes. Of course, cloud providers not offering Redis was the intention of the license change from the Redis point of view. So mission accomplished for Redis.

But the flip side of course is that if you want to deploy on standard infrastructure rather than self hosting Redis, Valkey is now the easy, low risk path that probably should be the default for most companies that target AWS, Azure, GCP, etc. Same with Elasticsearch vs. Opensearch and a few other products where the community forked because of license changes.

Mentioning Elasticsearch because I know people in both communities and I'm deeply familiar with the stack. A few years on, Opensearch has taken a lot of the momentum from Elasticsearch.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#45
post #31

Earlier quoted context omitted.

I use PHP. None of the language tools or constructs available to me are adequate. https://blog.codinghorror.com/the-php-singularity/

And you want to embed Redis inside PHP as a solution?? That’s nuts.

Where else could they store their serialized PHP data structures? (just kidding)

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#47

Earlier quoted context omitted.

Probably because Redis gives you a very well-defined/understood set of rich data structures with built-in behavior like TTL, atomic operations, eviction, and persistence. These things are otherwise usually scattered across native types, helper classes, or entirely separate libraries.

It doesn’t seem like the right tool for the job, though. Aren’t your own programming language’s constructs much more well-defined / understood ?

Redis has some pretty useful primitive that many languages don't:

- HyperLogLog, bloom filter, other probabilistic data structures

- Geospatial operations on stored points and polygons

- Expiring keys, for creating caches

These aren't in most standard libraries, and the Redis implementations tend to be fast, robust and well understood.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#48
post #3

While I love Redis as a versatile tool for external data structures, it's still lacking in two areas IMHO: One, it would be cool to be able to embed it, similar to sqlite, directly into applications. Two, the HA story is so much more complicated than it should be. I totally acknowledge that concurrency and distributed computing is hard, but it should not require reading heaps of documentation and understanding two en…

What would be the point of embedding Redis into an application? What's the advantage of using Redis over using the builtin (or third party) data structures of the language the application is developed in? I'm asking as a non-webdev who never quite got what Redis actually does, but would love to learn.

To me the thing I like about Redis is that it gives you a storage engine very suitable for caches; it handles TTLs and memory pressure, as well as built-in serialization with the ability to get better performance by allowing for some data loss. At the same time, many users will be deploying small programs to individual machines. If you could just have Redis be embedded this would make it very operationally simple: no additional daemons and a single file to backup if you want to.

It would also be useful because of the ability to switch modalities. When running a multi node service, you can use Redis to share data between nodes and use Redis pubsub as a communication bus. If you wanted to support a simple single node configuration too, then it wouldn't need to be a special case, it could just go through the same mechanism but with an embedded Redis instance.

It's pretty similar to SQLite: being able to embed more or less a complete storage engine into your app can be very convenient and powerful.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#49
post #3

While I love Redis as a versatile tool for external data structures, it's still lacking in two areas IMHO: One, it would be cool to be able to embed it, similar to sqlite, directly into applications. Two, the HA story is so much more complicated than it should be. I totally acknowledge that concurrency and distributed computing is hard, but it should not require reading heaps of documentation and understanding two en…

> One, it would be cool to be able to embed it, similar to sqlite, directly into applications.

I've found myself wanting this on several occasions too. I.e. wanting all my rust backend processes (k8s pods) to have some minimal shared state, without having to spin up a Redis cluster. I've talked to Claude about it a couple of times, and it descends into something like, "you gotta use Raft or CRDTs, and pick 2 out of 3 from CAP". Which honestly seems pretty fair, and indicates to me that I'm dreaming for something magical.

Nonetheless, it is nice to hear someone else asking for this. If this is indeed feasible (even if simple/limited), then I'd be interested to try it.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#50
> Rate limiting is one of the most common Redis use cases. Traditionally, users implemented rate limiters using server-side Lua scripts combined with client logic. In Redis 8.8, we introduce a window counter rate limiter (by @raffertyyu, together with the Redis team).

I had a look for this and it turns out it's slightly mis-described there - it's not a window counter, it's a "GCRA (Generic Cell Rate Algorithm)" - a leaky bucket algorithm. Code here: https://github.com/redis/redis/blob/unstable/src/gcra.c

The code comments say it was heavily influenced by https://github.com/brandur/redis-cell by Brandur Leach.

It's a neat algorithm (I just learned about it today) - it only needs to store a single integer for each rate-limited key, which is the "Theoretical Arrival Time" when the bucket would next be empty.

Post reply on HN