Streams: a new general purpose data structure in Redis

Post by itaifrenkel » Wed, Oct 04, 2017, 8:02 AM UTC

manigandham wrote: post #137

itaifrenkel wrote: post #118

The consumer groups proposal breaks the FIFO abstraction of a stream by allowing multiple clients to process a single stream. Have you considered adding a semantic layer inside streams that allows each client to consume a substream? In effect the stream becomes multiplexed substreams. If substreams makes the design too complex... have you considered server side stream 403 semantics? When a stream is manually deprecat…

Don't use consumer groups and every client will get a complete copy of the stream. What is broken with that?

Post by itaifrenkel » Wed, Oct 04, 2017, 8:18 PM UTC

manigandham wrote: post #152

itaifrenkel wrote: post #151

Earlier quoted context omitted.

The use case I'm referring to is when the client must be sharded to avoid CPU or network or disk bottlenecks.

Then that's exactly what consumer groups help with but it sounds like you want partitioning then - which is exactly what Kafka does but with a little more automation. Run multiple Redis instances and use a simple hash based on whatever key you want to route messages and get the throughput you need. It's probably never going to be part of the core Redis logic but should be possible as a module to do the routing when u…