Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streams: a new general purpose data structure in Redis

antirez.com

Post reply on HN
Print view View original

91–100 of 154 posts

Re: Streams: a new general purpose data structure in Redis

#91

Post by takeda »

erulabs wrote: post #66
hueving wrote: post #50

Earlier quoted context omitted.

But that doesn't explain why Kafka has any minimum the output required. Does it have usability issues? A good tool should be able to be used at any scale.

Kafka has very poor tooling in my experience (a folder full of fairly buggy bash scripts...), and due to ZooKeeper requires a lot of operational care. For example, it's extremely easy to destroy a Kafka cluster by bringing a new, empty ZK server online with newer but incorrect data in its volume. ZK will happily trash the entire cluster thinking it has new instructions. So network isolation is key, which, while obvio…

> For example, it's extremely easy to destroy a Kafka cluster by bringing a new, empty ZK server online with newer but incorrect data in its volume. ZK will happily trash the entire cluster thinking it has new instructions.

How does that happen? I mean a new, empty ZK server with never data than the rest of the cluster?

Also, please note that ZK is not meant to be a database, but a coordination service, it's guarantee is to have all nodes being always in consistent state and neither of its nodes allow to make any changes if there's no quorum. So if a new node somehow has more recent data with higher serial number it's expected that remaining nodes will sync to that.

Re: Streams: a new general purpose data structure in Redis

#92

Post by bonkabonka »

hakanito wrote: post #53
luhn wrote: post #19

I'm very excited about this. I've been eying HTTP EventSource for a while now, but there hasn't been a good solution for the backend broker. Kafka is overkill and Amazon Kinesis' pricing isn't viable if you have lots of topics. This fills the need perfectly and Redis is already part of my stack.

Just wanted to chip in and say HTTP EventSource has been really nice to work with, we've been using it in production for 1+ year

Which polyfill do use for IE/Edge?

Re: Streams: a new general purpose data structure in Redis

#93

Post by derefr »

takeda wrote: post #69
derefr wrote: post #63

Earlier quoted context omitted.

I'd agree in general, but Redis is architected in a way where these features aren't really new "layers", but rather just horizontal modular additions. A new data structure + associated commands being supported in Redis is like, say... a new filesystem being supported in the Linux kernel. It's a few files that you could just avoid compiling in if you didn't want them, and which add code-paths that are never run unless…

> I'd agree in general, but Redis is architected in a way where these features aren't really new "layers", but rather just horizontal modular additions. > A new data structure + associated commands being supported in Redis is like, say... a new filesystem being supported in the Linux kernel. It's a few files that you could just avoid compiling in if you didn't want them, and which add code-paths that are never run un…

I made that very specific comparison for a reason: most "plugin systems" for software (like Apache) act sort of like audio VSTs—they can insert themselves anywhere in the "processing chain" of a request, making following the logic more complex.

Redis modules, like filesystem drivers, are comparatively simple: they register a set of commands that they respond to, and each such command is handled exclusively by that module when received. Less like Apache plugins, more like scripts in a cgi-bin. You don't need to know what else is in the cgi-bin besides your own script, because everything there is entirely independent.

Re: Streams: a new general purpose data structure in Redis

#94

Post by manigandham »

jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

Thought/discussion usually leads to quality. Time has nothing to do with it.

Re: Streams: a new general purpose data structure in Redis

#95

Post by philsnow »

antirez wrote: post #70
philsnow wrote: post #68

Is there a reason to choose millis as the granularity instead of micros or nanos? Is it because there's a stronger expectation of machines in a cluster agreeing on what milli it is "now" vs the other granularities? I'm kind of thrown by the idea of putting the timestamp / stream-id in the XADD command, I would have thought the server would assign that, since one of the strengths of redis's single threaded nature is c…

Hello, the stream implementation does not need for the different servers (for instance master and its slaves) to agree about the time. Simply the server that receives the XADD command will generate the ID (and the time part of the ID) to attach to the item. All the other participants in the replication will accept the same ID, because clients will use " " to specify the ID, while the command is rewritten to slaves wi…

> Redis will anyway not accept any ID which is smaller than the current top-item ID.

I was reading on mobile earlier and maybe missed this point, excellent. I also didn't realize that clients would use (star) to specify the ID and that the receiving server turns that into an actual ID before replicating it / AOFing it.

> But being the time the one produced by the local host, after a failover the IDs are generated by another host. Milliseconds can still more or less match with good time synchronization, but nanoseconds? So it's like if this additional precision will be just used to store non-valid info.

I think most failovers necessarily take longer than a millisecond so any resolution smaller than millis would _probably_ be okay, but yeah this is not a compelling reason to switch to micros/nanos. My suggestion to switch to micros/nanos was more to try to reduce the number of collisions requiring the server to de-dup / assign sequential sub-epoch numbers to events arriving during the same server tick. I guess that's not a big issue though.

Thanks for the reply, Salvatore. Redis is one of my favorite codebases and projects.

Re: Streams: a new general purpose data structure in Redis

#96

Post by neduma »

jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

I heard that ruby was developed like that. It had lot of `freeze & thaw cycles` of good years before finally it got released.

Re: Streams: a new general purpose data structure in Redis

#97

Post by lsiebert »

antirez wrote: post #82
agumonkey wrote: post #71

Earlier quoted context omitted.

What were or would be your prefered alternative syntax ? `:` ?

Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)

I think a mostly vertical symbol is better, ":" or "|" is preferably to "_" or "-".

Re: Streams: a new general purpose data structure in Redis

#98

Post by lsiebert »

rakoo wrote: post #84
unkown-unknowns wrote: post #80

Earlier quoted context omitted.

It might be useful to be able to query by server time regardless of whether your client clock is in sync. You retrieve some set of data and the next time you can ask the server to give you everything newer than x, where x was the highest time stamp you got from the server previously.

Yes exactly, you want to ask what is newer than x, where x is the last event you're aware of, but you don't really care about the date and time in that case. If you just store the last id given by redis Streams naively then you don't even care that they're timestamps; at that point my question is, why even bother with the distinction. Just ask for everything after x and be done with it.

Redis also has TIME to get the current server time with milliseconds and the unix time stamp. I'm reasonably sure that's what's being used to get the first part of the ID anyway.

Re: Streams: a new general purpose data structure in Redis

#99

Post by dvt »

manigandham wrote: post #94
jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

Thought/discussion usually leads to quality. Time has nothing to do with it.

Of course it does: thought/discussion take time.

Re: Streams: a new general purpose data structure in Redis

#100

Post by erulabs »

I have one follow up question - is TTL a planned feature? Being able to set a TTL on the _stream itself_ and -also- on the messages would be extremely nice. While MAXLEN prevents a queue from being extremely large, I also want to remove "stale" data after a configurable time period.

Use case: A log of network latencies, where a user might currently `XREAD` with a timestamp 10 minutes in the past, would be able to save on memory usage by expiring log entries > 10 minutes, and then being able to `XREAD STREAMS strm 0` and let Redis (and therefore the infrastructure, not my code, manage data retention).

Also, how does this work re: evictions? Say a node is at max memory, will entire _streams_ be evicted, or (I hope) the oldest messages in the LRUed or LFUed queues.

Post reply on HN