Streams: a new general purpose data structure in Redis • Hacker News Board

Re: Streams: a new general purpose data structure in Redis

#11

Post by poorman » Mon, Oct 02, 2017, 5:23 PM UTC

> A final important thing to note about XRANGE is that, given that we receive the IDs in the reply, and the immediately successive ID is trivially obtained just incrementing the sequence part of the ID, it is possible to use XRANGE to incrementally iterate the whole stream, receiving for every call the specified number of elements.

Love it. No need for an expensive SCAN command.

Re: Streams: a new general purpose data structure in Redis

#12

Post by noncoml » Mon, Oct 02, 2017, 5:25 PM UTC

notamy wrote: post #8

After reading this I'm still not sure I understand. Time-series data I can see, but is there a use-case outside of this?

Poor-man’s Kafka?

Re: Streams: a new general purpose data structure in Redis

#13

Post by orixilus » Mon, Oct 02, 2017, 5:27 PM UTC

there's video[1] at the end explaining this new feature

[1] https://www.youtube.com/watch?v=ELDzy9lCFHQ

Re: Streams: a new general purpose data structure in Redis

#14

Post by femto113 » Mon, Oct 02, 2017, 5:29 PM UTC

I have a confusion about ID structure/format:

   The ID is composed of two parts: a millisecond time and a
   sequence number.  The number after the dot is the
   sequence number, and is used in order to distinguish
   entries added in the same millisecond.

Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadvertently) will yield the wrong order (as would sorting them lexicographically). If so it seems like something other than a decimal point would be a better separator (colon perhaps).

Re: Streams: a new general purpose data structure in Redis

#15

Post by zedpm » Mon, Oct 02, 2017, 5:30 PM UTC

hyades wrote: post #6

Under what circumstances would one prefer Redis streams over Kafka and vice versa?

One that immediately comes to mind is cases where Kafka is overkill. Kafka is a great tool, but there's a lot of overhead in setting up and maintaining it (e.g. Zookeeper), so if your throughput needs are low, it's a poor fit. Spinning up a Redis server is dead simple, and if you're already using Redis for other things, then there's no need to bring an additional tool into the mix.

Re: Streams: a new general purpose data structure in Redis

#16

Post by sterex » Mon, Oct 02, 2017, 5:38 PM UTC

femto113 wrote: post #14

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…

Perhaps decoupling the as (# x 0.01) would solve the problem right?

Not sure if that's what they do, but it isn't very complex.

Re: Streams: a new general purpose data structure in Redis

#17

Post by agacera » Mon, Oct 02, 2017, 5:38 PM UTC

hyades wrote: post #6

Under what circumstances would one prefer Redis streams over Kafka and vice versa?

I can think in some circumtances: 1 - You already have a Redis infrastructure and don't wanna or don't have resources to deploy a full Kafka infrastructure (3 kafka brokers + 3 zookeeper nodes)

2 - Kafka clients are not available (or are poorly available) for every programming language. Redis has a simpler protocol, so it has more/and better clients available and even if you use an exotic language, it is easy to write a client to it (well... easier than Kafka)

3 - Kafka AFAIK does not have any internal cache implementation, so every read is served from disk (+ page cache). This means that Redis Streams will (probably) perform much better for use cases when the consumers need to fetch data from old offsets.

edit: added reason number 3.

Re: Streams: a new general purpose data structure in Redis

#18

Post by manigandham » Mon, Oct 02, 2017, 5:39 PM UTC

hyades wrote: post #6

Under what circumstances would one prefer Redis streams over Kafka and vice versa?

Simplicity. Redis 4.0 with the rxlists module [1] provides a fast queue system with full persistence. Unless you need replay, Redis is often easier and has plenty of throughput. This streams feature now solves the replay disadvantage.

1. http://redismodules.com/modules/rxlists/

Re: Streams: a new general purpose data structure in Redis

#19

Post by luhn » Mon, Oct 02, 2017, 5:41 PM UTC

I'm very excited about this. I've been eying HTTP EventSource for a while now, but there hasn't been a good solution for the backend broker. Kafka is overkill and Amazon Kinesis' pricing isn't viable if you have lots of topics. This fills the need perfectly and Redis is already part of my stack.

Re: Streams: a new general purpose data structure in Redis

#20

Post by antirez » Mon, Oct 02, 2017, 5:41 PM UTC

femto113 wrote: post #14

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…

Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.