Redis streams as a pure data structure • Hacker News Board

Re: Redis streams as a pure data structure

#51

Post by aarbor989 » Sat, Mar 23, 2019, 11:50 PM UTC

manigandham wrote: post #49

opportune wrote: post #39

Earlier quoted context omitted.

From my experience, Kafka has the best api for handling read-once, distributed streams. Almost every other streaming solution, like Redis in this case, has a non-ideal or non-existent way to coordinate stream consumers in a way that prevents double-reads. And lots of streaming applications need to ensure read-once (think about what a double read ends up as - maybe a twice-sent message, or a duplicate metric), so I'm…

How so? Kafka only accepts offsets which are meant for batches of items or even an entire partition. This means a single item not being processed within that batch requires your own code to compensate. It's the weakest of all the messaging models. Per-message acknowledgement is an advancement. Redis requires manual lookups for unacknowledged items but you can also use Apache Pulsar for a more scalable distributed dis…

I’m not sure what you mean by “only accepts offsets which are meant for batches” but with Kafka the offsets are per-partition and you have to flexibility to control exactly when that offset is marked as processed. In our systems we always used manual offset committing and would only commit an offset once processing of the message has completed successfully to ensure both at-most-once and seamless failover.

Re: Redis streams as a pure data structure

#52

Post by nicois » Sun, Mar 24, 2019, 12:32 AM UTC

I threw together a few words here about how we are using Streams combined with Sorted Sets to "upgrade" legacy databases to streams of data. Not revolutionary, but it could be interesting to some people. I can write more, if there's any demand: http://nicois.github.io/posts/databases-to-streams/

Re: Redis streams as a pure data structure

#53

Post by manigandham » Sun, Mar 24, 2019, 4:56 AM UTC

aarbor989 wrote: post #51

manigandham wrote: post #49

Earlier quoted context omitted.

How so? Kafka only accepts offsets which are meant for batches of items or even an entire partition. This means a single item not being processed within that batch requires your own code to compensate. It's the weakest of all the messaging models. Per-message acknowledgement is an advancement. Redis requires manual lookups for unacknowledged items but you can also use Apache Pulsar for a more scalable distributed dis…

I’m not sure what you mean by “only accepts offsets which are meant for batches” but with Kafka the offsets are per-partition and you have to flexibility to control exactly when that offset is marked as processed. In our systems we always used manual offset committing and would only commit an offset once processing of the message has completed successfully to ensure both at-most-once and seamless failover.

Offsets are a marker that says everything before it (in that partition) is processed. This creates 2 issues:

1) Your application must coordinate and make sure that everything up to that offset is indeed processed successfully. 2) You application must stop if it encounters an error (because it can't commit an offset greater than that item) or handle it separately by logging to another topic, database, etc.

Other systems like Redis, Pulsar, Google PubSub provide per-message acknowledgement to allow items to be individually processed without blocking other forward progress.

Re: Redis streams as a pure data structure

#54

Post by senderista » Sun, Mar 24, 2019, 8:59 AM UTC

aarbor989 wrote: post #51

manigandham wrote: post #49

Earlier quoted context omitted.

How so? Kafka only accepts offsets which are meant for batches of items or even an entire partition. This means a single item not being processed within that batch requires your own code to compensate. It's the weakest of all the messaging models. Per-message acknowledgement is an advancement. Redis requires manual lookups for unacknowledged items but you can also use Apache Pulsar for a more scalable distributed dis…

I’m not sure what you mean by “only accepts offsets which are meant for batches” but with Kafka the offsets are per-partition and you have to flexibility to control exactly when that offset is marked as processed. In our systems we always used manual offset committing and would only commit an offset once processing of the message has completed successfully to ensure both at-most-once and seamless failover.

That doesn’t sound like “at-most-once”. What if your consumer crashed after processing the message but before committing the offset?

Re: Redis streams as a pure data structure

#55

Post by aarbor989 » Sun, Mar 24, 2019, 4:46 PM UTC

senderista wrote: post #54

aarbor989 wrote: post #51

Earlier quoted context omitted.

I’m not sure what you mean by “only accepts offsets which are meant for batches” but with Kafka the offsets are per-partition and you have to flexibility to control exactly when that offset is marked as processed. In our systems we always used manual offset committing and would only commit an offset once processing of the message has completed successfully to ensure both at-most-once and seamless failover.

That doesn’t sound like “at-most-once”. What if your consumer crashed after processing the message but before committing the offset?

This is a fair thing to point out. For us the “at-most-once” guarantee was based on committing. Luckily our processing was idempotent so in the rare case of the above scenario it wouldn’t cause any duplication

Re: Redis streams as a pure data structure

#56

Post by aarbor989 » Sun, Mar 24, 2019, 5:26 PM UTC

manigandham wrote: post #53

aarbor989 wrote: post #51

Earlier quoted context omitted.

I’m not sure what you mean by “only accepts offsets which are meant for batches” but with Kafka the offsets are per-partition and you have to flexibility to control exactly when that offset is marked as processed. In our systems we always used manual offset committing and would only commit an offset once processing of the message has completed successfully to ensure both at-most-once and seamless failover.

Offsets are a marker that says everything before it (in that partition) is processed. This creates 2 issues: 1) Your application must coordinate and make sure that everything up to that offset is indeed processed successfully. 2) You application must stop if it encounters an error (because it can't commit an offset greater than that item) or handle it separately by logging to another topic, database, etc. Other syste…

Ah, I see what you’re saying now. #2 was always a pain to deal with, but I think other systems have similar problems. Other messaging systems deal with this with things like dead letter queues etc, but no matter what you use for message processing you will need some specialized logic to handle records which can’t be processed normally. In Kafka, you can raise an exception for the offset and then move on. When dealing with the exception, you can seek directly to the record offset and take it from there.

For #1, any application which has an in-order requirement would suffer from this problem. I worked with event processing systems so we never really had to worry about this, since each event was independent. However, there were instances where we would need to track state for certain objects getting processed to make sure all of their child objects were also processed. For this we would use an external store with a short TTL since the lifetime of the object during processing would only be a few minutes.

All-in-all it just comes down to what your app’s requirements are. I don’t think Kafka is meant to replace every pub sub service out there, but definitely has some great use cases.