Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis 8.8: New array data structure, rate limiter, performance improvements

redis.io

Post reply on HN
Print view View original

71–80 of 118 posts

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#71

Post by yxhuvud »

amtamt wrote: post #4
9dev wrote: post #3

While I love Redis as a versatile tool for external data structures, it's still lacking in two areas IMHO: One, it would be cool to be able to embed it, similar to sqlite, directly into applications. Two, the HA story is so much more complicated than it should be. I totally acknowledge that concurrency and distributed computing is hard, but it should not require reading heaps of documentation and understanding two en…

Genuinely interested why we need HA in redis, just not read round robin from multiple non-HA instances? Redis (and memcache) are memory caches and should be treated like that, not like highly consistent distributed session store.

Redis have many use cases, and acting as a cache is only one of them. One very common usage is as a backend for background worker jobs. That can need HA.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#72

Post by s_trumpet »

jchw wrote: post #68
0x457 wrote: post #67

Earlier quoted context omitted.

Well, if you have a single instance than using language libraries and structures will be better in most cases. If you use multiple nodes, then you probably want your redis lifecycle not be tied to application lifecycle.

I am not aware of an in-process alternative similar to what Redis offers.

Mnesia, if you’re using Erlang or Elixir.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#73

Post by n_e »

tossandthrow wrote: post #64
9dev wrote: post #30

Earlier quoted context omitted.

It’s not. Imagine a web app that stores your user information in a session store, mapped by your cookie-provided session ID. Your web app searches redis 1 for the session id, but since that key is on redis 2, the lookup fails and the application thinks there is no such session, and rejects the request. Now you could solve this specific case by sharding by prefix, or by querying all instances, but then you still do no…

I don't think you understand what HA means. The app would look up in both databases. If it exists in any, there would be a session. Thisnis strictly different from partitioning which I think you are mixing it up with. Paritioning is for performance not HA

> The app would look up in both databases. If it exists in any, there would be a session.

And if you find the session with differing values in both databases, how do you know which one is up-to-date?

You need an algorithm to pick which data is right, such as electing a master instance.

And that brings us back to the original discussion: to manage sessions (unlike caches) in a highly available way, you need to setup HA (or reimplement it, which obviously is a bad idea). You can't read round robin from multiple non-HA instances.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#74

Post by tossandthrow »

n_e wrote: post #73
tossandthrow wrote: post #64

Earlier quoted context omitted.

I don't think you understand what HA means. The app would look up in both databases. If it exists in any, there would be a session. Thisnis strictly different from partitioning which I think you are mixing it up with. Paritioning is for performance not HA

> The app would look up in both databases. If it exists in any, there would be a session. And if you find the session with differing values in both databases, how do you know which one is up-to-date? You need an algorithm to pick which data is right, such as electing a master instance. And that brings us back to the original discussion: to manage sessions (unlike caches) in a highly available way, you need to setup H…

Yes, you are pointing out exactly how HA is difficult.

There is a whole slew of downstream things you need to take into consideration.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#75

Post by rsalus »

freakynit wrote: post #35
stingraycharles wrote: post #25

Earlier quoted context omitted.

It doesn’t seem like the right tool for the job, though. Aren’t your own programming language’s constructs much more well-defined / understood ?

Language's own native data-structures are generally much more capable and vast. 99%+ developers use only a very limited set of those capabilities. This approach packages those most used ones into a nice, consistent DSL. It's similar in effect to what busybox does to shell utilities, though the motives are different.

agreed but depends on then language. for instance, the .NET equivalent (MemoryCache) is pretty poor.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#77

Post by williamdclt »

adamcharnock wrote: post #49
9dev wrote: post #3

While I love Redis as a versatile tool for external data structures, it's still lacking in two areas IMHO: One, it would be cool to be able to embed it, similar to sqlite, directly into applications. Two, the HA story is so much more complicated than it should be. I totally acknowledge that concurrency and distributed computing is hard, but it should not require reading heaps of documentation and understanding two en…

> One, it would be cool to be able to embed it, similar to sqlite, directly into applications. I've found myself wanting this on several occasions too. I.e. wanting all my rust backend processes (k8s pods) to have some minimal shared state, without having to spin up a Redis cluster. I've talked to Claude about it a couple of times, and it descends into something like, "you gotta use Raft or CRDTs, and pick 2 out of 3…

I don't know if that'll make you feel any better but yeah, you're indeed asking for the impossible! You need consensus between your nodes that store state _somehow_, either these nodes are Redis and it does that for you, or these nodes are your pods and you need to do consensus yourself (zookeeper might help, but you're definitely in "complicated stuff" territory).

Spinning up an in-memory (no persistence) Redis cluster in your k8s should be easy enough, hopefully?

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#78

Post by simonw »

0x457 wrote: post #69
simonw wrote: post #47

Earlier quoted context omitted.

Redis has some pretty useful primitive that many languages don't: - HyperLogLog, bloom filter, other probabilistic data structures - Geospatial operations on stored points and polygons - Expiring keys, for creating caches These aren't in most standard libraries, and the Redis implementations tend to be fast, robust and well understood.

Can you name a single language that can talk to redis and doesn't have these in a form of a library that integrates with an app better than mystical embedded redis? Every language you can talk to redis most likely has a library to do that, and it probably works much better with the rest of application than "embedded redis". If it doesn't, it probably has C-FFI and there is "fast, robust and well understood" implement…

Sure. But if Redis was embeddable you'd get a robust C-FFI style implementation of those data structures which has been tested a lot more than some random library that has almost no existing users or active maintenance.

(I'm not personally sold on embedded Redis myself, but the question was "Aren’t your own programming language’s constructs much more well-defined / understood?")

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#79

Post by b33j0r »

willempienaar wrote: post #58
simonw wrote: post #50

> Rate limiting is one of the most common Redis use cases. Traditionally, users implemented rate limiters using server-side Lua scripts combined with client logic. In Redis 8.8, we introduce a window counter rate limiter (by @raffertyyu, together with the Redis team). I had a look for this and it turns out it's slightly mis-described there - it's not a window counter, it's a "GCRA (Generic Cell Rate Algorithm)" - a l…

Also, the “cell” in Generic Cell Rate Algorithm is an ATM cell. GCRA is 1990s telecom, the scheduling algorithm ATM switches used to check that 53-byte cells were arriving on the wire at the agreed rate.

Asynchronous Transfer Mode (I had to look it up too.)

https://en.wikipedia.org/wiki/Asynchronous_Transfer_Mode

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#80

Post by jchw »

s_trumpet wrote: post #72
jchw wrote: post #68

Earlier quoted context omitted.

I am not aware of an in-process alternative similar to what Redis offers.

Mnesia, if you’re using Erlang or Elixir.

Unfortunately I have never really used Erlang outside of deploying RabbitMQ. I mostly use Go, Rust, Python, sometimes C/C++.

However, Mnesia seems like it is quite a bit more of a complete distributed database engine than Redis. To me the nicest thing about Redis is just the convenience of what it offers: very fast data structures, serialized, optimized (at least by default) for cases where speed is more important than durability. It is simple on many levels and somewhat constrained in scope. Mnesia seems to be aiming more generally in the distributed database category.

So how do you feel they compare?

Post reply on HN