Live data from Hacker News

A multithreaded fork of Redis that is faster

docs.keydb.dev

61–70 of 164 posts

Re: A multithreaded fork of Redis that is faster

#61

You went to the effort of forking Redis but didn’t bother to fix the glaring lack of TLS support? What is the fucking point of a replicated KV store if the connections between nodes aren’t encrypted.

We have it in Redis 6, ETA for RC1 is end of year. However we did it the right way with a socket abstraction layer, one of the reasons it took so long.

Re: A multithreaded fork of Redis that is faster

#62
post #33

On 32 cores? Doesn't seem like particularly great scaling.

In contrast running N instances scales linearly. That's why Redis multi threading just attempts to get the low hanging fruit of the write calls. For serious scalability it's better to orchestrate multiple instances in our vision.

Re: A multithreaded fork of Redis that is faster

#63
post #35

Earlier quoted context omitted.

But isn't the point of Redis is to be used as a caching layer instead of persistent layer

But what if you could have the speed of RAM-based storage, and still have persistence? Sounds pretty appealing to me, personally.

You already get this with either Redis or Keydb. The problem is that to avoid the disk being the bottleneck you must write to RAM, then later to the disk (you can't wait for confirmation that it was written) which means that even though a key may successfully be written to the DB, it hasn't been confirmed to persist.

Basically redis can't necessarily guarantee data safety - which is perfectly fine for it's normal use case as a caching layer and data you don't necessarily care if you lose (for example rate limiting relies on tracking the number of requests/s within say an hour - if you lose the data you don't really care). User sessions too, worst case people have to log in again, meh whatever.

Re: A multithreaded fork of Redis that is faster

#64
post #61

You went to the effort of forking Redis but didn’t bother to fix the glaring lack of TLS support? What is the fucking point of a replicated KV store if the connections between nodes aren’t encrypted.

We have it in Redis 6, ETA for RC1 is end of year. However we did it the right way with a socket abstraction layer, one of the reasons it took so long.

Is there any relevant documentation/discussion available?

Re: A multithreaded fork of Redis that is faster

#65
post #57
post #20

Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-us…

The idle CPU might very well be due to redis being single threaded. It might be hitting is limits and still show low CPU utilization. Also, you didn't mention that, but I've seen this happening often, you should never use redis as a primary data store if your data is important.

You'd expect a 1/n where n is number of cores available utilization in the case of redis being CPU bound.

I guess I don't know how redis deals with the network, but I'd assume that it handles concurrent requests. If not, then I guess that's the case where you'd see less than 1/n cpu utilization and it could still be CPU bound.

Re: A multithreaded fork of Redis that is faster

#67

Earlier quoted context omitted.

Facebook just published a blog about moving petabytes per hour. I read it, I find the design interesting. I don't need it. But if I do, it's good to know that it's possible, that someone has solved, to have a reference and to possibly use their tool if it's free. So relax, I for one welcome whatever folks are working on out there. If it's true, we can learn from it and apply it in order domains too.

> Facebook just published a blog about moving petabytes per hour. For the curious: https://engineering.fb.com/data-infrastructure/scribe/ Edit: HN thread: https://news.ycombinator.com/item?id=21181982

You should post that.

Re: A multithreaded fork of Redis that is faster

#68
post #38
post #20

Any time these "much faster than Redis" databases come up, the sysadmin in me wonders how many people have had actual performance limitation issues with Redis. I've seen Redis servers handle hundreds of GB of traffic per hour. I've worked at companies where Aerospike and others are proposed as replacements for Redis because "they're faster" - and I point out the 98% idle CPUs on the Redis server, and the near-100%-us…

If redis isn't multi-threaded the more cores and threads you have, the less CPU usage the machine would show, even if Redis is being stressed.

Except if you monitor per CPU usage, as you should.

Re: A multithreaded fork of Redis that is faster

#69
post #61

You went to the effort of forking Redis but didn’t bother to fix the glaring lack of TLS support? What is the fucking point of a replicated KV store if the connections between nodes aren’t encrypted.

We have it in Redis 6, ETA for RC1 is end of year. However we did it the right way with a socket abstraction layer, one of the reasons it took so long.

[deleted]

Re: A multithreaded fork of Redis that is faster

#70
post #61

Earlier quoted context omitted.

We have it in Redis 6, ETA for RC1 is end of year. However we did it the right way with a socket abstraction layer, one of the reasons it took so long.

Is there any relevant documentation/discussion available?

Yes, the implementation is in this PR: https://github.com/antirez/redis/pull/6236
Post reply on HN