Live data from Hacker News

Streams: a new general purpose data structure in Redis

antirez.com

131–140 of 154 posts

Re: Streams: a new general purpose data structure in Redis

#131

Slightly off-topic, but could the blog be adjust a little for mobile reading? #content { max-width: 800px; } // replaces width: 800px seems to do the job, and also behaves better in narrow desktop browser windows.

Done hopefully... EDIT: undone, makes things much worse on Android :-)

Re: Streams: a new general purpose data structure in Redis

#132

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

> This is how actual quality software that will stand the test of time gets designed and made .. No not really. It's called waterfall and we had it for decades. And it was largely abandoned in favour of agile development because despite architects spending months designing something there was always something that was left out. Or the scope changed during those months. Or millions of other things that change whilst y…

You missed the part where the definition of the problem was being percolated during this process. Agile won't save you from not understanding the problem you are trying to solve.

Re: Streams: a new general purpose data structure in Redis

#133
post #82

Earlier quoted context omitted.

Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)

I think a mostly vertical symbol is better, ":" or "|" is preferably to "_" or "-".

I agree. Perhaps a forward-slash (/) to denote subordination, e.g. 1507035873/11

Re: Streams: a new general purpose data structure in Redis

#134

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

> This is how actual quality software that will stand the test of time gets designed and made .. No not really. It's called waterfall and we had it for decades. And it was largely abandoned in favour of agile development because despite architects spending months designing something there was always something that was left out. Or the scope changed during those months. Or millions of other things that change whilst y…

I disagree, I think "waterfall" (at least in stereotype) would be more like coming up with the specifications in advance and then _sticking to them_ regardless of new information from the world or from the experience of developing the software. That's not the story the OP told.

I also agree with the person who responded that there is a difference between building products, and building tools for building products -- building shared libraries/code/tools may require different approaches. One of the things I've noticed about shared code, is it's _much harder_ to fix rethought decisions or change paths. "Backwards incompatibility" doesn't even exist as a thing in your local app really, at least you can always at least theoretically global search and replace when you change APIs.

I don't think we've figured out any magic bullet process for developing software. It's definitely not the stereotype of "waterfall", but I don't think the stereotype of "agile" is it either -- at _least_ when it comes to shared library/tools. And I'm absolutely convinced that those shared tools which do well were almost always designed carefully and intentionally with understanding of the domain, not just slapped together with stimulus-response.

Re: Streams: a new general purpose data structure in Redis

#135
post #99

Earlier quoted context omitted.

Thought/discussion usually leads to quality. Time has nothing to do with it.

Of course it does: thought/discussion take time.

It seems people are confused. Of course everything takes physical time, but slowly thinking about something isn't any objective sign of better outcomes.

Re: Streams: a new general purpose data structure in Redis

#136
post #20

Earlier quoted context omitted.

Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.

I vote that you use a dash instead.

It turns out that's what it will be [0], as dash retains the ability to easily copy the whole identifier in most terminals.

[0]: https://github.com/antirez/redis/commit/1189d90d749c84e98424...

Re: Streams: a new general purpose data structure in Redis

#137

The consumer groups proposal breaks the FIFO abstraction of a stream by allowing multiple clients to process a single stream. Have you considered adding a semantic layer inside streams that allows each client to consume a substream? In effect the stream becomes multiplexed substreams. If substreams makes the design too complex... have you considered server side stream 403 semantics? When a stream is manually deprecat…

Don't use consumer groups and every client will get a complete copy of the stream. What is broken with that?

Re: Streams: a new general purpose data structure in Redis

#138
post #15

Earlier quoted context omitted.

One that immediately comes to mind is cases where Kafka is overkill. Kafka is a great tool, but there's a lot of overhead in setting up and maintaining it (e.g. Zookeeper), so if your throughput needs are low, it's a poor fit. Spinning up a Redis server is dead simple, and if you're already using Redis for other things, then there's no need to bring an additional tool into the mix.

Genuine question - Why does everyone seem to think running a zookeeper cluster is so hard? You can run it on three small VMs and basically forget about it. We didn't have any zookeeper experience at my last startup before we started using it for Kafka and we used a very simple puppet module to install it on three instances in each of our AWS regions. It really never gave us many problems in the several years since. A…

I'm not sure about it being "so hard", but it's extra stuff to deploy, maintain, monitor, and pay for. Very small teams benefit from keeping things small and simple. Again, if you actually need Kafka, then it's worth it. If you just need something similar, but can deal with the limitations of redis streams, then it's an easy choice.

Re: Streams: a new general purpose data structure in Redis

#139
This may actually solve an issue I was about to tackle, which is high-speed notification delivery. Currently I was going to do a nasty wrestle of PUB/SUB with Lists and blocking keys to try and get a cluster of message processing servers to digest notifications when they get added to the "queue". My purpose in this is notifications, as in actual push notifications for mobile/web/etc. This seems like it fits perfectly with what I had planned. Especially since this ensures message delivery instead of fire-and-forget as you mentioned. As a question though... is the XACK commands working in your current branch? Since that is key to my usage of ensuring message consumption.

Re: Streams: a new general purpose data structure in Redis

#140

This may actually solve an issue I was about to tackle, which is high-speed notification delivery. Currently I was going to do a nasty wrestle of PUB/SUB with Lists and blocking keys to try and get a cluster of message processing servers to digest notifications when they get added to the "queue". My purpose in this is notifications, as in actual push notifications for mobile/web/etc. This seems like it fits perfectly…

Essentially, I was going down this route if anyone's curious....

http://code.flickr.net/2012/12/12/highly-available-real-time...

Post reply on HN