Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Redis Streams development

antirez.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: An update on Redis Streams development

#41

Post by manigandham »

jadbox wrote: post #38
antirez wrote: post #15

Earlier quoted context omitted.

Thanks for your interesting POV on the Redis/Kafka intersection on this. What could be interesting is that while Redis Streams are certainly totally inspired from Kafka streams, it's just a conceptual thing, so the two things can act in very different ways in practice. For instance just imagine that I never touched a Kafka system in my life, never used it, don't even know the API, I only read all the documentation th…

What level of guarantees will Redis Streams have during power failure? Can I configure if events are close to immediately persistent to disk VS served from memory and occasionally paged to disk?

Redis streams are just another data type and follow the same persistence settings that have been available to Redis: https://redis.io/topics/persistence

You can use either RDB: snapshots of the entire dataset at some interval and/or AOF: logging of all changes fsync on every write, every second or leaving it to the OS.

Re: An update on Redis Streams development

#42

Post by nicois »

From the blog post, it sounds like the RDB format will break. Is there any means to upgrade from the current unstable RDB format to the v5 one? Obviously there are no consumer groups in unstable now, so in theory reading from a v4 RDB should not be hard..

Re: An update on Redis Streams development

#43

Post by DerBesserWisser »

antirez wrote: post #17
pselbert wrote: post #16

I must admit, for the past couple months I've been digging for status info about Redis Streams. They would fit a use case we have perfectly, but we use cloud providers for Redis so manual compilation with modules isn't possible. Really pleased to see that I'm not the only one digging for info and that work is ongoing.

Thank you, I believe it's my fault that many potential users remained wondering. Sometimes I forget that the world is not inside Twitter... and I should instead blog more and tweet less, both for myself, because writing a blog post gives me much more sense to accomplish something, and for people interested in Redis that will find information more readily, and also searching via Google and so forth.

Your blog posts are wonderfully written.

Re: An update on Redis Streams development

#44

Post by mdomans »

antirez wrote: post #28
mdomans wrote: post #27

Here's a question from an old Redis hater (the note is important, since my question is going to be slightly biased - I disagree with a lot of core decisions behind Redis): How is this going to be different from Kafka? And I don't mean implementation details, because these are always fun read. Kafka is on the market for ~7 years, during which it has proven to be oh-so-fast and pretty durable. Oh, and while I'm at it.…

So yes, that's it. The Redis hidden agenda was to compete with Python (to be honest not a big secret... you can see that Redis works internally as an interpreter in a pretty obvious way), and now that you uncovered it, I'm going to say it aloud: we are going to exit in a few months with a new package system we are working at for 5 years at this point, based on the blockchain (proof of installation), which will kill N…

Chapeau bas. But apart from my rather obnoxious joking - how the whole Stream and GIL interoperate. I mean, this is technically a complex problem.

Re: An update on Redis Streams development

#45

Post by mdomans »

kahlonel wrote: post #29
mdomans wrote: post #27

Here's a question from an old Redis hater (the note is important, since my question is going to be slightly biased - I disagree with a lot of core decisions behind Redis): How is this going to be different from Kafka? And I don't mean implementation details, because these are always fun read. Kafka is on the market for ~7 years, during which it has proven to be oh-so-fast and pretty durable. Oh, and while I'm at it.…

How can you compare a programming language and a memory store? How are they even remotely comparable? Redis needs a syntax to exchange data with clients in a human-readable way. Honest question: Why do you hate Redis?

It was meant as a joke.

Alas, I have been more than once bitten by poorly implemented GIL and in my personal experience any GIL is problematic in a high-throughput environment. It simplifies the problem of implementation of environment or API for the price of introducing global lock into the system, thus eliminating any possibility of lockless design. And we know it's doable.

I spent years trying to work around GIL in production environments and we have the tools about now to do it reliably. We have on_commit in Django, we have Celery with proper support for mechanisms like chords and a myriad of others.

I do get that Redis has a different approach to many things. But here's the deal - every time I had to rely on Redis as a critical component, there was a problem. Either there was that "one gotcha" in the config or I didn't understood something or else.

A while ago I was a part of discussion with the author of Redis on using Redis as cache. And quite some people, not only me, noted they had bad experience and their benchmarks didn't show Redis is better than Memcached. And, I think that's ok. Unless you pose your product as a competitor to memcached. Would I use memcached-db? No, it's a bad idea.

My problem is that Redis originally was a key value store with datastructures. And that's a great idea and not that many services let you model datastructures in a distributed way. I actually implemented a distributed system of progress control for a very critical piece of architecture at my current workplace. I'm a Redis user since 2014 in regard to commercial products.

My point of view is probably largely skewed but I don't understand and I never seen that explained why Redis tries to do everything. We have battle tested Kafka for streams, ElasticSearch and Solr for searching and Memcached for caching. Can't we get a super reliable data store that supports data structures? A data store that is _able_ to utilize the fact we have more than one core?

I know my view is unpopular, but I don't see addition of GIL as a genius idea. I simply don't see a coherent direction for Redis development.

BTW: is there a benchmark comparing latest Redis without the GIL and the latest with GIL?

Re: An update on Redis Streams development

#46

Post by mdomans »

ehsanu1 wrote: post #31
kahlonel wrote: post #29

Earlier quoted context omitted.

How can you compare a programming language and a memory store? How are they even remotely comparable? Redis needs a syntax to exchange data with clients in a human-readable way. Honest question: Why do you hate Redis?

Had to re-read a few times, but the comparison just seems like a poorly worded attempt at a joke/insult.

Also as a pinch-of-a-salt in the whole sugary environment of HN where every new feature is cool by definition. For example, what are the shortcomings of Kafka compared to Redis Streams?

Re: An update on Redis Streams development

#47

Post by itamarhaber »

antirez wrote: post #28
mdomans wrote: post #27

Here's a question from an old Redis hater (the note is important, since my question is going to be slightly biased - I disagree with a lot of core decisions behind Redis): How is this going to be different from Kafka? And I don't mean implementation details, because these are always fun read. Kafka is on the market for ~7 years, during which it has proven to be oh-so-fast and pretty durable. Oh, and while I'm at it.…

So yes, that's it. The Redis hidden agenda was to compete with Python (to be honest not a big secret... you can see that Redis works internally as an interpreter in a pretty obvious way), and now that you uncovered it, I'm going to say it aloud: we are going to exit in a few months with a new package system we are working at for 5 years at this point, based on the blockchain (proof of installation), which will kill N…

lol

Re: An update on Redis Streams development

#48

Post by mdomans »

kahlonel wrote: post #29
mdomans wrote: post #27

Here's a question from an old Redis hater (the note is important, since my question is going to be slightly biased - I disagree with a lot of core decisions behind Redis): How is this going to be different from Kafka? And I don't mean implementation details, because these are always fun read. Kafka is on the market for ~7 years, during which it has proven to be oh-so-fast and pretty durable. Oh, and while I'm at it.…

How can you compare a programming language and a memory store? How are they even remotely comparable? Redis needs a syntax to exchange data with clients in a human-readable way. Honest question: Why do you hate Redis?

And an extra note when someone says "it's hard to implement ... without locks and so". Look at Firefox. I dumped it's bushy tail around 7 years ago when switching to the apple-scented world for good.

I came crawling back. Why? Because people at Mozilla spent their time working on their product, finally coming to agreement that the fact your product is OS doesn't mean you can cut your standards in half focusing on features you like instead the ones that actually make it better.

Here's the link in case someone missed it https://hacks.mozilla.org/2017/11/entering-the-quantum-era-h.... I'm not a Mozilla fanboy, to be frank for a while foxy browser scored lower on my "software like scale" than Redis.

And they won me over. In 5 minutes. And it's been proven time and time again that performance is a feature.

p.s. Now that I write it I realize that for certain reasons I should just try to write the damn thing and instead of writing pointy comments let antirez rip my software apart :)

Post reply on HN