Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streams: a new general purpose data structure in Redis

antirez.com

Post reply on HN
Print view View original

81–90 of 154 posts

Re: Streams: a new general purpose data structure in Redis

#81

Post by mioelnir »

agacera wrote: post #17
hyades wrote: post #6

Under what circumstances would one prefer Redis streams over Kafka and vice versa?

I can think in some circumtances: 1 - You already have a Redis infrastructure and don't wanna or don't have resources to deploy a full Kafka infrastructure (3 kafka brokers + 3 zookeeper nodes) 2 - Kafka clients are not available (or are poorly available) for every programming language. Redis has a simpler protocol, so it has more/and better clients available and even if you use an exotic language, it is easy to writ…

Regarding point 3, unless your system is under massive memory pressure, no caught-up Kafka consumer should be serviced from disk. Old offsets that are flushed out of memory because you do not have it obviously are served from disk, with essentially linear reads of disk blocks (of consecutive logical addresses if flush sizes are large enough that then can end up on disk in any number of ways, depending on how much the firmware lies, I know) of the requested file.

I really can not see how Redis is going to perform "much better" reading from disk once the entries are no longer in RAM. At that point both Kafka and Redis have to read from disk, and you either have the IOPS to serve all the lagging consumers or you don't. Maybe you have enough of them to service 1 or 2 concurrent reads, maybe 10-12. But for the same messages counts, sizes and concurrent consumers, your workload will become IOPS bound rather fast.

Note: "much better" to me implies 10x+ better, not "my C library read() is 2.3% better than your Java".

Re: Streams: a new general purpose data structure in Redis

#82

Post by antirez »

agumonkey wrote: post #71
antirez wrote: post #20

Earlier quoted context omitted.

Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.

What were or would be your prefered alternative syntax ? `:` ?

Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)

Re: Streams: a new general purpose data structure in Redis

#83

Post by takeda »

chicagobuss wrote: post #34
zedpm wrote: post #15

Earlier quoted context omitted.

One that immediately comes to mind is cases where Kafka is overkill. Kafka is a great tool, but there's a lot of overhead in setting up and maintaining it (e.g. Zookeeper), so if your throughput needs are low, it's a poor fit. Spinning up a Redis server is dead simple, and if you're already using Redis for other things, then there's no need to bring an additional tool into the mix.

Genuine question - Why does everyone seem to think running a zookeeper cluster is so hard? You can run it on three small VMs and basically forget about it. We didn't have any zookeeper experience at my last startup before we started using it for Kafka and we used a very simple puppet module to install it on three instances in each of our AWS regions. It really never gave us many problems in the several years since. A…

For some reason Zookeeper is unjustly seem as uncool technology. I even seen it being blamed for issues that it had nothing to do with.

People say that setting ZK cluster is a huge issue, yet they don't see a problem spinning etcd, or sentry nodes in case of redis.

When I learned about ZK I was skeptic, didn't like that it was written in Java, but ZK proved to be extremely robust.

Re: Streams: a new general purpose data structure in Redis

#84

Post by rakoo »

unkown-unknowns wrote: post #80
rakoo wrote: post #56

Earlier quoted context omitted.

Why keep the separation at all ? Are clients expected to be able to query for a given timestamp precisely ? Because then you get all the problems with clock synchronization, especially given that the Streams' clock is monotonic and I'd expect clients' clock to not be

It might be useful to be able to query by server time regardless of whether your client clock is in sync. You retrieve some set of data and the next time you can ask the server to give you everything newer than x, where x was the highest time stamp you got from the server previously.

Yes exactly, you want to ask what is newer than x, where x is the last event you're aware of, but you don't really care about the date and time in that case. If you just store the last id given by redis Streams naively then you don't even care that they're timestamps; at that point my question is, why even bother with the distinction. Just ask for everything after x and be done with it.

Re: Streams: a new general purpose data structure in Redis

#85

Post by mioelnir »

erulabs wrote: post #66
hueving wrote: post #50

Earlier quoted context omitted.

But that doesn't explain why Kafka has any minimum the output required. Does it have usability issues? A good tool should be able to be used at any scale.

Kafka has very poor tooling in my experience (a folder full of fairly buggy bash scripts...), and due to ZooKeeper requires a lot of operational care. For example, it's extremely easy to destroy a Kafka cluster by bringing a new, empty ZK server online with newer but incorrect data in its volume. ZK will happily trash the entire cluster thinking it has new instructions. So network isolation is key, which, while obvio…

As someone that has both Kafka and Redis in use without issue, for years, (and is about to replace a lot of misused Redis instances with Kafka) I really fail to follow your points.

So, a Zookeeper cluster can't survive accidentally injecting just the right malicious data that will make it keel over. I'm sorry, how do you accidentally achieve that? Do you also accidentally configure your Redis Sentinel to replicate from /dev/null?

As a matter of fact, this announcement comes at a very inopportune time for me. antirez had the epiphany of reading on IRC about replicated logs instead of looking at the opening paragraphs of the Kafka documentation, and all the Redis evangelists at my job will now try to shoe-horn the wrong usecase back into Redis because Redis!1cos(0)!. Sigh.

Re: Streams: a new general purpose data structure in Redis

#86

Post by antirez »

Goopplesoft wrote: post #74

What sort of compression do the blocks undergo? E.g. does periodicity of the timeseries help reduce the the space of the 64/128bit timestamp? Gorilla[1] style compression would be great, although it'd likely make sub block level range queries tough. [1] http://www.vldb.org/pvldb/vol8/p1816-teller.pdf

Hello, yes IDs are delta compressed so they use actually just a few bytes per entry (often just 2) instead of 16.

Re: Streams: a new general purpose data structure in Redis

#88

Post by jrochkind1 »

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I accumulated talking with people about this upcoming addition to Redis."

I've been thinking about this sort of thing for a while, wanting to maybe call it "slow code" (like "slow food"). This is how actual quality software that will stand the test of time gets designed and made, _slowly_, _carefully_, _intentionally_, with thought and discussion and feedback and reconsideration. And always based on understanding the domain and the existing software you are building upon. Not jumping from problem to a PR to a merge. (And _usually_ by one person, sometimes a couple/several working together, _rarely_ by committee).

Re: Streams: a new general purpose data structure in Redis

#89

Post by baileymiller182 »

luhn wrote: post #19

I'm very excited about this. I've been eying HTTP EventSource for a while now, but there hasn't been a good solution for the backend broker. Kafka is overkill and Amazon Kinesis' pricing isn't viable if you have lots of topics. This fills the need perfectly and Redis is already part of my stack.

Check out nchan.io, an nginx module that does everything you need to connected EventSource to redis backed pub/sub.
Post reply on HN