Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Redis Streams development

antirez.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: An update on Redis Streams development

#31

Post by ehsanu1 »

kahlonel wrote: post #29
mdomans wrote: post #27

Here's a question from an old Redis hater (the note is important, since my question is going to be slightly biased - I disagree with a lot of core decisions behind Redis): How is this going to be different from Kafka? And I don't mean implementation details, because these are always fun read. Kafka is on the market for ~7 years, during which it has proven to be oh-so-fast and pretty durable. Oh, and while I'm at it.…

How can you compare a programming language and a memory store? How are they even remotely comparable? Redis needs a syntax to exchange data with clients in a human-readable way. Honest question: Why do you hate Redis?

Had to re-read a few times, but the comparison just seems like a poorly worded attempt at a joke/insult.

Re: An update on Redis Streams development

#32

Post by manigandham »

Great to see. We actually ended up using Redis for our event stream after trying Kafka and the rest. Using the list extension module multiple list pop functions lets us get to 2gbps of throughput on a single redis node with AOF persistence. Using streams would make things even simpler and faster.

Re: An update on Redis Streams development

#33

Post by luhn »

viperscape wrote: post #24
luhn wrote: post #6

Earlier quoted context omitted.

Redis Pubsub is fire-and-forget, so if you aren't listening when a message is fired, you'll never receive it. Redis Streams store messages, so you can connect and read all the messages since you last checked. It's a similar model to Kafka.

is it strictly in-mem? for instance if redis is restarted is there an option for unread streams to be saved with redis persistence

The built-in persistence mechanisms should be compatible with Streams.

Re: An update on Redis Streams development

#34

Post by somethingsimple »

manigandham wrote: post #32

Great to see. We actually ended up using Redis for our event stream after trying Kafka and the rest. Using the list extension module multiple list pop functions lets us get to 2gbps of throughput on a single redis node with AOF persistence. Using streams would make things even simpler and faster.

Did you have issues with Kafka? Curious to hear because I’m about to try and start using it for something at work.

Re: An update on Redis Streams development

#35

Post by manigandham »

somethingsimple wrote: post #34
manigandham wrote: post #32

Great to see. We actually ended up using Redis for our event stream after trying Kafka and the rest. Using the list extension module multiple list pop functions lets us get to 2gbps of throughput on a single redis node with AOF persistence. Using streams would make things even simpler and faster.

Did you have issues with Kafka? Curious to hear because I’m about to try and start using it for something at work.

It's just much more work to install and maintain Kafka, and it has issues with load balancing and recovery due to the design tying cluster ownership to partition data. With AOF persistence and a replica, Redis is durable enough for us and extremely fast with no maintenance.

If you absolutely need Kafka then it's still a good option, although I'd recommend looking at Apache Pulsar [1] for a better design. It separates storage and compute for better performance and scalability while giving you features like per-message acknowledgements.

1. https://pulsar.apache.org

Re: An update on Redis Streams development

#36

Post by epaulson »

Are there any examples of good web APIs that offer something like a unified log as an abstraction? I'm not looking for systems, but actual companies that have some kind of "streaming" data feed where you can (re)connect to an endpoint and say "give me everything from [logical] timestamp X". Ideally one where you stay connected and get longpoll SSE/WebSockets/MQTT-style streaming responses.

I kind of want the opposite of webhooks.

Re: An update on Redis Streams development

#37

Post by jkarneges »

epaulson wrote: post #36

Are there any examples of good web APIs that offer something like a unified log as an abstraction? I'm not looking for systems, but actual companies that have some kind of "streaming" data feed where you can (re)connect to an endpoint and say "give me everything from [logical] timestamp X". Ideally one where you stay connected and get longpoll SSE/WebSockets/MQTT-style streaming responses. I kind of want the opposite…

There are certainly "client oriented" (as opposed to webhooks) push APIs out there. However, such APIs that let you specify a starting position to read from are rare.

Sometimes APIs will give you tokens to use for resumption (e.g. SSE event IDs, or any long-polling API), but typically these are for a time-limited session rather than a stateless query against any point in a long-lived log.

Years ago, services like Friendfeed, Livefyre, and Convore had stateless long-polling APIs that returned a log of data, I believe. These kinds of APIs seem to have fallen out of fashion, though. There are still stateless long-polling APIs, but most of the ones I'm aware of don't return logs of data. For example, Dropbox and Box will let you query for a change notification against a starting position, but then you have to fetch the actual data separately.

That said, just because streaming APIs that let you set a starting position are rare doesn't mean they're impossible to make. My company (https://fanout.io) has built tools to help with this.

Edit: since you asked for a real example, Superfeedr is one such API: https://documentation.superfeedr.com/subscribers.html#stream...

Re: An update on Redis Streams development

#38

Post by jadbox »

antirez wrote: post #15
neovintage wrote: post #12

I believe Redis has a great developer experience. It’s easy to get set up and use. When I see the work on Redis streams, I think its going bring a much better getting started experience for developers that want to start using evented architectures. This might be a turning point where we see more developers and applications utilizing those types of architectures. In the end, this might give Apache Kafka a run for its…

Thanks for your interesting POV on the Redis/Kafka intersection on this. What could be interesting is that while Redis Streams are certainly totally inspired from Kafka streams, it's just a conceptual thing, so the two things can act in very different ways in practice. For instance just imagine that I never touched a Kafka system in my life, never used it, don't even know the API, I only read all the documentation th…

What level of guarantees will Redis Streams have during power failure? Can I configure if events are close to immediately persistent to disk VS served from memory and occasionally paged to disk?

Re: An update on Redis Streams development

#39

Post by misterbowfinger »

antirez wrote: post #15
neovintage wrote: post #12

I believe Redis has a great developer experience. It’s easy to get set up and use. When I see the work on Redis streams, I think its going bring a much better getting started experience for developers that want to start using evented architectures. This might be a turning point where we see more developers and applications utilizing those types of architectures. In the end, this might give Apache Kafka a run for its…

Thanks for your interesting POV on the Redis/Kafka intersection on this. What could be interesting is that while Redis Streams are certainly totally inspired from Kafka streams, it's just a conceptual thing, so the two things can act in very different ways in practice. For instance just imagine that I never touched a Kafka system in my life, never used it, don't even know the API, I only read all the documentation th…

> that is to create an API for developers thinking like I'm designing an iPhone

Albeit my request is quite selfish, I really would love to hear/read more about your thoughts on designing APIs. My experience using Redis has been excellent and I'd to be able to replicate that sense of design in the systems I build as well.

Re: An update on Redis Streams development

#40

Post by ec109685 »

antirez wrote: post #13
acidus wrote: post #5

I wonder if this impacts the plan of releasing Disque as a plugin in Redis 4.2. I always thought that Disque could have a great impact in the field of job queues.

Hello, no change... The original plan was: 4.0 (done) -> Streams back ported to 4.0 (Work in progress) -> 4.2 (or 5.0) with Disque + Cluster improvements + Modules improvements, ... It's just a renaming: 4.0 + Streams backported is now called 5.0 What was to be 4.2 is going to be called 6.0 Why I'm choosing to go for integer numbers? Because I believe that things like 4.2 should be for minor improvements, mostly oper…

Love that you are using sem version properly. Too many folks couple marketing reasons into their versioning schemes as opposed to simply, “did I break backwards compatibility?”
Post reply on HN