I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…
Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.
Streams: a new general purpose data structure in Redis
71–80 of 154 posts
Re: Streams: a new general purpose data structure in Redis
#72Re: Streams: a new general purpose data structure in Redis
#73Earlier quoted context omitted.
re: client support - I dunno, this seems like a pretty comprehensive list to me? I mean, there's even a rust client: https://cwiki.apache.org/confluence/display/KAFKA/Clients
Confluent only officially supports the Java client (and now has a Python, Go and .Net clients as well that I didn't know) and it is really recommended to use the client with the same version of broker due to protocol incompatibilities. Most Kafka client implementations are open-source projects of their own, this is also true for most redis clients implementations, but again: Kafka protocol is much more complicated th…
Re: Streams: a new general purpose data structure in Redis
#74Re: Streams: a new general purpose data structure in Redis
#75It's been a long time since I looked into this: is there now a way to configure a cluster of Redis instances such that you won't lose messages on node failure? If not, all the nice at-least-once delivery (or "effectively once" when you add message dedupe) you get with something like Kafka/Kinesis/GCP PubSub is gone. If not, either people's messages don't matter /that/ much (which is fine, just not great for most of m…
Hello, the streams have basically the same characteristics as any other Redis data structure, that is, from the POV of a local node, you can configure strong persistence on disk, but on node failures, you have basically different tunable amount of best effort consistency, it means that you cannot guarantee no messages are lost. So basically this means that you can: 1. Use the default asynchronous replication, and liv…
Re: Streams: a new general purpose data structure in Redis
#76Projects tend to gain more and more functionality to match the new workloads they're being used to accomplish, and it must certainly be a difficult decision for project visionaries. Do I listen to my users and implement features that will solve their new woes, but in return accept increased complexity and higher learning barriers? Complexity sucks, but it's even harder to say no to users in pain. I wonder if this ope…
Another data point, streams are if not completely, at least 70% done as we talk, and yet `git log --stat unstable..streams` reports:
12 files changed, 1323 insertions(+), 10 deletions(-)
We'll end at 2000 lines of code I guess more or less. Just to say, Redis is at a complexity scale very far from many other things we are used to these days.
Re: Streams: a new general purpose data structure in Redis
#77Re: Streams: a new general purpose data structure in Redis
#78Earlier quoted context omitted.
The dot doesn't make that a decimal, any more than it makes IP addresses or version numbers decimals. As for treating them as decimals inadvertently, well, hopefully client libraries will expose IDs as pairs of integers, not as strings. If users convert them into strings and then back into meaningless pseudo-decimals, well, great, we'll have an entertaining post about someone's outage to read.
This is a terrible attitude to have when responding to an obvious potential UX confusion, particularly when it will only come up in edge cases (>10 per millisecond).
Re: Streams: a new general purpose data structure in Redis
#79I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…
Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.
Re: Streams: a new general purpose data structure in Redis
#80Earlier quoted context omitted.
Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.
Why keep the separation at all ? Are clients expected to be able to query for a given timestamp precisely ? Because then you get all the problems with clock synchronization, especially given that the Streams' clock is monotonic and I'd expect clients' clock to not be