Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis 8.8: New array data structure, rate limiter, performance improvements

redis.io

Post reply on HN
Print view View original

101–110 of 118 posts

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#101

Post by 9dev »

trumpdong wrote: post #63
9dev wrote: post #59

Earlier quoted context omitted.

Redis is a real database. If I wasn’t convinced it could retain data I hand it, I wouldn’t use it in the first place. Just because it works for your use case right now doesn’t mean there isn’t room for improvements to support others too.

> Redis is a real database. Oh good, then you don't need to do any of the stuff that you suggested to do

These two concerns are not mutually exclusive, the kind of database or data stored within it doesn't give any availability guarantees on its own. Even a single Postgres instance, which I suppose fits your understanding of a real database, is a single point of failure and not a highly available setup: If your database server goes down, clients get errors and the database is thus unavailable.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#102

Post by 9dev »

dvkashapov wrote: post #99
lukaslalinsky wrote: post #34

Earlier quoted context omitted.

With the amount of problems I had using Redis Sentinel, I really wish there was another way. On multiple occasions, with completely different deployments, it got itself into a non-repairable state where the only option was to drop it and setup the replicas manually. I was hoping someone would do a Patroni-like project for Redis, but I've not found it yet. I've moved all persistent data to PostgreSQL and use a number…

I suggest you to take a look at rdsync ( https://github.com/yandex/rdsync ), exactly what you want: Patroni-like high-availability tool for Valkey/Redis. Uses ZooKeeper for external coordination. We use it in our large deployment and with a couple patches you will forged about the need to take manual actions to resolve broken states.

At which scale would you recommend this over a Sentinel setup?

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#103

Post by dvkashapov »

9dev wrote: post #102
dvkashapov wrote: post #99

Earlier quoted context omitted.

I suggest you to take a look at rdsync ( https://github.com/yandex/rdsync ), exactly what you want: Patroni-like high-availability tool for Valkey/Redis. Uses ZooKeeper for external coordination. We use it in our large deployment and with a couple patches you will forged about the need to take manual actions to resolve broken states.

At which scale would you recommend this over a Sentinel setup?

To be honest - at any scale, this really does help me not wake up at night to fix broken states by hand as sometimes on-call engineer. Although note that rdsync is mainly for Valkey up to 9.1, there were Redis patches for 7.2 (last BSD version).

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#104

Post by jaapz »

flaghacker wrote: post #13
9dev wrote: post #3

While I love Redis as a versatile tool for external data structures, it's still lacking in two areas IMHO: One, it would be cool to be able to embed it, similar to sqlite, directly into applications. Two, the HA story is so much more complicated than it should be. I totally acknowledge that concurrency and distributed computing is hard, but it should not require reading heaps of documentation and understanding two en…

What would be the point of embedding Redis into an application? What's the advantage of using Redis over using the builtin (or third party) data structures of the language the application is developed in? I'm asking as a non-webdev who never quite got what Redis actually does, but would love to learn.

I have 15 processes handling API requests over HTTP. I want rate limiting, so users don't DoS my API. I need shared state across these processes to store information about rate limits for each client key. I store those in redis.

Or queues, caching, pub/sub... Redis can do a lot, it's really (and i mean really) fast, very easy to get up and running and the protocol is pretty simple.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#105

Post by amtamt »

9dev wrote: post #59
trumpdong wrote: post #56

Earlier quoted context omitted.

If you consider it important, you have to store it in a real database. No buts. If you don't consider it important, sharded redis works fine.

Redis is a real database. If I wasn’t convinced it could retain data I hand it, I wouldn’t use it in the first place. Just because it works for your use case right now doesn’t mean there isn’t room for improvements to support others too.

> Redis is a real database

No, redis is a memory cache, with some ACID like features bolt on.

Even real databases have hard time maintaining consistency across nodes. CAP is a real constraint.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#106

Post by amtamt »

9dev wrote: post #42
olavgg wrote: post #40

Earlier quoted context omitted.

But that is his point. If you cannot find the session id in redis, you login again. If your Redis server crash, you start a new one and everyone just login again. No data is lost.

Sure the data is lost. A session commonly holds arbitrary state, and even if it’s just the login information. This is ridiculous.

No two processes can guarantee data consistency unless using shared memory with some kind of locking on update. And given two servers don't share memory, two processes running on these servers can not guarantee consistency either.

To put the simple terms... App writes to node-A, node-A (/process on node-A) crashes before change is synced from node-A to node-B, data is lost.

This is true for redis and true for postgresql/ mysql or any similar database. Difference between redis and a "database" is that database protects against this problem by writing change to durable storage before telling app that write is successful. Redis

So if one wants to have a consistent session storage, one should use a proper database or use AOF redis persistense with single node (https://chatgpt.com/s/t_6a24ab818e2881918db959cec8d8cc2d)

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#107

Post by 9dev »

amtamt wrote: post #106
9dev wrote: post #42

Earlier quoted context omitted.

Sure the data is lost. A session commonly holds arbitrary state, and even if it’s just the login information. This is ridiculous.

No two processes can guarantee data consistency unless using shared memory with some kind of locking on update. And given two servers don't share memory, two processes running on these servers can not guarantee consistency either. To put the simple terms... App writes to node-A, node-A (/process on node-A) crashes before change is synced from node-A to node-B, data is lost. This is true for redis and true for postgre…

First up, if I wanted to talk to a machine, I would've asked one myself.

Then, I don't understand your point really: Yes, the CAP theorem is a thing. There are compromise solutions available however to enable highly available data storage. Some of them for Redis too, but they are more complicated than those for other database engines. Which is the point of this discussion.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#108

Post by amtamt »

9dev wrote: post #107
amtamt wrote: post #106

Earlier quoted context omitted.

No two processes can guarantee data consistency unless using shared memory with some kind of locking on update. And given two servers don't share memory, two processes running on these servers can not guarantee consistency either. To put the simple terms... App writes to node-A, node-A (/process on node-A) crashes before change is synced from node-A to node-B, data is lost. This is true for redis and true for postgre…

First up, if I wanted to talk to a machine, I would've asked one myself. Then, I don't understand your point really: Yes, the CAP theorem is a thing. There are compromise solutions available however to enable highly available data storage. Some of them for Redis too, but they are more complicated than those for other database engines. Which is the point of this discussion.

Point is... with AOF and RDB enabled, and wait command used in sane manner, one can get reasonable consistency with a significant speed tradeoff and increase in application complexity. So if consistent cache is needed, one can have that with some compromises, but then probably one could use a database straight away.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#109

Post by 9dev »

amtamt wrote: post #108
9dev wrote: post #107

Earlier quoted context omitted.

First up, if I wanted to talk to a machine, I would've asked one myself. Then, I don't understand your point really: Yes, the CAP theorem is a thing. There are compromise solutions available however to enable highly available data storage. Some of them for Redis too, but they are more complicated than those for other database engines. Which is the point of this discussion.

Point is... with AOF and RDB enabled, and wait command used in sane manner, one can get reasonable consistency with a significant speed tradeoff and increase in application complexity. So if consistent cache is needed, one can have that with some compromises, but then probably one could use a database straight away.

Again: Redis is a database, not just a cache - it doesn’t care if you store ephemeral cache artifacts or customer records within it. Redis doesn’t pose any constraints on the type of data. Inversely, you can use Postgres as a semi or fully consistent cache.

And yes, what you’re saying is technically correct, even a well-tuned single node doesn’t solve the availability problem: if it goes down, you have an outage. To avoid that, you need multiple instances to provide the same data, avoiding downtime if one of the nodes breaks eventually.

Re: Redis 8.8: New array data structure, rate limiter, performance improvements

#110

Post by amtamt »

9dev wrote: post #109
amtamt wrote: post #108

Earlier quoted context omitted.

Point is... with AOF and RDB enabled, and wait command used in sane manner, one can get reasonable consistency with a significant speed tradeoff and increase in application complexity. So if consistent cache is needed, one can have that with some compromises, but then probably one could use a database straight away.

Again: Redis is a database, not just a cache - it doesn’t care if you store ephemeral cache artifacts or customer records within it. Redis doesn’t pose any constraints on the type of data. Inversely, you can use Postgres as a semi or fully consistent cache. And yes, what you’re saying is technically correct, even a well-tuned single node doesn’t solve the availability problem: if it goes down, you have an outage. To…

If redis is a database, then by

> doesn’t pose any constraints on the type of data.

logic, raw disk is also a database. One just need to add block level replication to other nodes to build a replicated/ HA database.

We may not agree, but anything not providing transactions across logically related multiple data read/ update operations is not a database.

> multiple instances to provide the same data

is easy and done by a bunch of software out there, but

> multiple instances to provide the same data on non-shared memory computers, with consistency

is a really hard problem, and no one has been able to solve it yet without introducing other problems to be considered (giving up on fast performance being one of the most visible one) by architects/ developers.

Post reply on HN