Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Redis Streams development

antirez.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: An update on Redis Streams development

#11

Post by itamarhaber »

acidus wrote: post #5

I wonder if this impacts the plan of releasing Disque as a plugin in Redis 4.2. I always thought that Disque could have a great impact in the field of job queues.

That's for Salvatore to decide, but the version is just an arbitrary choice. Assuming that Disque is coming as a Redis module, it may be even supported by v4.0. OTOH, I believe that implementing Disque will have implications on the Modules API, so these would have to be backported to a v4.2 (or alike) if it is decided so. In any case, this shouldn't have an impact on the Disque roadmap significantly IMHO.

Re: An update on Redis Streams development

#12

Post by neovintage »

I believe Redis has a great developer experience. It’s easy to get set up and use. When I see the work on Redis streams, I think its going bring a much better getting started experience for developers that want to start using evented architectures. This might be a turning point where we see more developers and applications utilizing those types of architectures. In the end, this might give Apache Kafka a run for its money, or not, who knows. I've tried using Apache Kafka and it can be a bear to set up given its dependencies and to administer.

Put another way, I think Kafka solved a problem for enterprises and was a tops down approach to the problem. Redis streams is a bottoms up approach to implementing evented architectures. Maybe there's room for both products in the market?

Re: An update on Redis Streams development

#13

Post by antirez »

acidus wrote: post #5

I wonder if this impacts the plan of releasing Disque as a plugin in Redis 4.2. I always thought that Disque could have a great impact in the field of job queues.

Hello, no change... The original plan was:

4.0 (done) ->

Streams back ported to 4.0 (Work in progress) ->

4.2 (or 5.0) with Disque + Cluster improvements + Modules improvements, ...

It's just a renaming:

4.0 + Streams backported is now called 5.0

What was to be 4.2 is going to be called 6.0

Why I'm choosing to go for integer numbers? Because I believe that things like 4.2 should be for minor improvements, mostly operational, but to add the first data structure after ages deserves 5.0, similarly to have a reshaped Redis Cluster + Disque deserves 6.0, and I get myself confused as user of other systems when they advance like 1.4, 2.3, 2.7, ... It's simpler to talk about Redis 4, Redis 5, Redis 6, ...

Re: An update on Redis Streams development

#14

Post by pselbert »

netgusto wrote: post #2

Could someone ELI5 what's a redis stream ? I thought that pub/sub mecanisms were some kind of a stream already.

This article from Brandur does a great job of explaining a use case for Redis Streams [0]. As part of that he begins by explaining what they are, how they work, and a bit about how they differ from Kafka.

[0]: https://brandur.org/redis-streams

Re: An update on Redis Streams development

#15

Post by antirez »

neovintage wrote: post #12

I believe Redis has a great developer experience. It’s easy to get set up and use. When I see the work on Redis streams, I think its going bring a much better getting started experience for developers that want to start using evented architectures. This might be a turning point where we see more developers and applications utilizing those types of architectures. In the end, this might give Apache Kafka a run for its…

Thanks for your interesting POV on the Redis/Kafka intersection on this. What could be interesting is that while Redis Streams are certainly totally inspired from Kafka streams, it's just a conceptual thing, so the two things can act in very different ways in practice.

For instance just imagine that I never touched a Kafka system in my life, never used it, don't even know the API, I only read all the documentation they have on the site about the design, to get the higher level picture and combine this with my own ideas about fixing the fact Redis was lacking a "log" data structure.

Pub/Sub + other data structures were not able to provide time series and streaming, but yet Redis streams remain an ADT (Abstract Data Structure), while Kafka is a tool to solve a very specific business case. So the applications have some intersection, but are also very very apart.

For instance you can create infinite small keys having streams, so Redis is good for many IoT usages where you receive data from many small devices. Redis Streams also stress on range queries, so that you can combine the time series with millisecond-range queries.

However, yes, the fact that I added also consumer groups is a way to put this "80% streaming" into a more usable streaming systems more similar to Kafka, for the use cases where:

1) The memory limits.

2) The speed.

3) The consistency guarantees of Redis make sense.

However at the same time, it was a great challenge and pleasure to do what I always try to do, that is to create an API for developers thinking like I'm designing an iPhone, and not some terrible engineering thing which does what it should but is terrible to use (I'm not referring to Kafka that I do not know). So I really hope that what you say "easy to setup and use" will be what developers will feel :-)

Re: An update on Redis Streams development

#16

Post by pselbert »

I must admit, for the past couple months I've been digging for status info about Redis Streams. They would fit a use case we have perfectly, but we use cloud providers for Redis so manual compilation with modules isn't possible.

Really pleased to see that I'm not the only one digging for info and that work is ongoing.

Re: An update on Redis Streams development

#17

Post by antirez »

pselbert wrote: post #16

I must admit, for the past couple months I've been digging for status info about Redis Streams. They would fit a use case we have perfectly, but we use cloud providers for Redis so manual compilation with modules isn't possible. Really pleased to see that I'm not the only one digging for info and that work is ongoing.

Thank you, I believe it's my fault that many potential users remained wondering. Sometimes I forget that the world is not inside Twitter... and I should instead blog more and tweet less, both for myself, because writing a blog post gives me much more sense to accomplish something, and for people interested in Redis that will find information more readily, and also searching via Google and so forth.

Re: An update on Redis Streams development

#18

Post by antirez »

nicolaslem wrote: post #10
acidus wrote: post #5

I wonder if this impacts the plan of releasing Disque as a plugin in Redis 4.2. I always thought that Disque could have a great impact in the field of job queues.

My feeling is that Redis itself is good enough for job queues so that there is not a huge pressure to improve it with Disque. I actually implemented yet another job framework[0] for fun in Python with Redis and it was a pleasure. Lua, pub/sub and atomic operations really go a long way! [0] https://github.com/NicolasLM/spinach

What I loved about Disque is that it was the magical distributed systems thing that everybody really want for messages safety, but done in a way totally transparent for the user, and it actually worked well enough that there are people keeping using the RC1 for ages... That's why I want to resurrect such a project. The fact that it is multi master, auto federated and auto balancing, and all the auto stuff happen without any user stress, was kinda a good thing. As a Redis module everything will be the same, but, without all the code duplication, because otherwise Disque is like a 5000 lines of code project or alike, if not less.

Re: An update on Redis Streams development

#20

Post by acidus »

antirez wrote: post #13
acidus wrote: post #5

I wonder if this impacts the plan of releasing Disque as a plugin in Redis 4.2. I always thought that Disque could have a great impact in the field of job queues.

Hello, no change... The original plan was: 4.0 (done) -> Streams back ported to 4.0 (Work in progress) -> 4.2 (or 5.0) with Disque + Cluster improvements + Modules improvements, ... It's just a renaming: 4.0 + Streams backported is now called 5.0 What was to be 4.2 is going to be called 6.0 Why I'm choosing to go for integer numbers? Because I believe that things like 4.2 should be for minor improvements, mostly oper…

That sounds like a good plan, thank you!
Post reply on HN