Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streams: a new general purpose data structure in Redis

antirez.com

Post reply on HN
Print view View original

101–110 of 154 posts

Re: Streams: a new general purpose data structure in Redis

#101

Post by lsiebert »

antirez wrote: post #87
silverwind wrote: post #77

The use of `+` and `-` in XRANGE seems inconsistent. Why not use `0` and `-1` like LRANGE?

Because they do not mean a position, but a special ID.

It seems that "$" is a special ID for the last message, as opposed to the last possible message.

I would humbly suggest that "^" would be a suitable symbol for the first message in a stream. ^ and $ are used in regex (and vim) in a similar way.

That way you could write "XREAD BLOCK 5000 STREAMS newstream ^" and get all the messages in a stream from the beginning, and then block until a new message comes in all with a single command. You would still be able to add a count if needed, to prevent client flooding.

Re: Streams: a new general purpose data structure in Redis

#102

Post by femto113 »

antirez wrote: post #82
agumonkey wrote: post #71

Earlier quoted context omitted.

What were or would be your prefered alternative syntax ? `:` ?

Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)

I'll probably eventually hate myself for even bringing this up but I can't help but notice the similarity between this ID structure and Version 1 (aka timestamp) UUIDs. While I wouldn't go as far as recommending that you fully adopt that form, it might be worth considering if you could make these IDs compatible with UUIDs by defining a canonical transform. The critical differences are:

- UUIDs use a different epoch (15 Oct 1582 vs 1 Jan 1970) - UUIDs count 100 ns blocks instead of ms - UUIDs include a 6 byte "node id" - UUIDs allow only up to 15 bits of "sequence"

I think that last one is the biggest deal, since as currently specced redis allows 64 bits of sequence, which is obviously much bigger than 15. The options I see are either up the time resolution used by redis, encode some of redis's sequence bits into the UUID's time bits, or just live with it as a limitation--in practice 2^15 is a lot of messages to get in a single millisecond (though in cases of clocks jumping back might not be too much).

You'd also need to come up with some thing for the node id, perhaps the first 6 bytes of a cluster node ID or similar.

Re: Streams: a new general purpose data structure in Redis

#103

Post by plasma »

Are there plans for a "unique count" over an XRANGE?

I currently use multiple Sorted Sets (one set every 5 minutes) and Union 30-60 worth to produce a "rolling window" of uniques.

I can see an alternative where I just request a unique count of elements within an XRANGE.

Re: Streams: a new general purpose data structure in Redis

#104

Post by digaozao »

jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

I like this kind of development also. But I don't think it is for everyone (the person has to keep thinking on the problem for long time).

A good talk about It, from the creator of clojure:

https://youtu.be/f84n5oFoZBc

Re: Streams: a new general purpose data structure in Redis

#105

Post by anyfoo »

lsiebert wrote: post #97
antirez wrote: post #82

Earlier quoted context omitted.

Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)

I think a mostly vertical symbol is better, ":" or "|" is preferably to "_" or "-".

Why is that?

Re: Streams: a new general purpose data structure in Redis

#106

Post by anyfoo »

twic wrote: post #23
femto113 wrote: post #14

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…

The dot doesn't make that a decimal, any more than it makes IP addresses or version numbers decimals. As for treating them as decimals inadvertently, well, hopefully client libraries will expose IDs as pairs of integers, not as strings. If users convert them into strings and then back into meaningless pseudo-decimals, well, great, we'll have an entertaining post about someone's outage to read.

It's not much of a problem for (v4) IPs, because they almost always consist of 4 numbers separated by a dot, making them immediately distinguishable from decimal numbers. If two-component IPs were common (they are sometimes seen in CIDR notation, but not often), the dot would have been an unfortunate separator choice as well.

For versions with only two components, I would argue that the dot can be confusing already.

Why use a separator that has the potential of confusion when there are several other choices with less potential?

Re: Streams: a new general purpose data structure in Redis

#107

Post by anyfoo »

ihattendorf wrote: post #31
femto113 wrote: post #14

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…

What about regions that treat the comma as a decimal separator? I agree with the other commenter that this is no different than using periods in IPv4 addresses.

As a member of such a region, I suspect that those regions a well aware that the prevalent notation throughout programming uses '.' as a decimal separator. IPv4 addresses usually have 4 components and do not have an inherently fractional unit as their first component.

Re: Streams: a new general purpose data structure in Redis

#108

Post by hiram112 »

erulabs wrote: post #66
hueving wrote: post #50

Earlier quoted context omitted.

But that doesn't explain why Kafka has any minimum the output required. Does it have usability issues? A good tool should be able to be used at any scale.

Kafka has very poor tooling in my experience (a folder full of fairly buggy bash scripts...), and due to ZooKeeper requires a lot of operational care. For example, it's extremely easy to destroy a Kafka cluster by bringing a new, empty ZK server online with newer but incorrect data in its volume. ZK will happily trash the entire cluster thinking it has new instructions. So network isolation is key, which, while obvio…

> ...in my experience it takes years and years and years for Java based infrastructure to become relatively stable & reliable (see ElasticSearch 5.0, or ask anyone who has been oncall for a Tomcat based application).

This is almost the exact opposite of how I determine what tools to use. If it's written in C or Java, I'm usually pretty confident that it is engineered by a team of experienced developers. Both because the languages are technically more difficult to use, and not as cool.

In contrast, if a tool is written with javascript (Node) or Ruby, and often times Python, I'm very hesitant.

In fact, this whole topic of streams has me wondering just how many developers out there are setting up clusters of Kafka or Redis or whatever is new and hip, when they could have saved themselves a huge amount of pain by using tried and true tools like JMS or ZeroMQ.

Most companies do NOT have a need for scaling like Netflix or LinkedIn, and I'm beginning to wonder if Kafka, Redis, etc are this year's version of NoSQL and MongoDB hype.

Re: Streams: a new general purpose data structure in Redis

#109

Post by erulabs »

hiram112 wrote: post #108
erulabs wrote: post #66

Earlier quoted context omitted.

Kafka has very poor tooling in my experience (a folder full of fairly buggy bash scripts...), and due to ZooKeeper requires a lot of operational care. For example, it's extremely easy to destroy a Kafka cluster by bringing a new, empty ZK server online with newer but incorrect data in its volume. ZK will happily trash the entire cluster thinking it has new instructions. So network isolation is key, which, while obvio…

> ...in my experience it takes years and years and years for Java based infrastructure to become relatively stable & reliable (see ElasticSearch 5.0, or ask anyone who has been oncall for a Tomcat based application). This is almost the exact opposite of how I determine what tools to use. If it's written in C or Java, I'm usually pretty confident that it is engineered by a team of experienced developers. Both because…

I should have clarified - web applications in Java, built by web developers, are not to be trusted. ES -usage-, not ES itself, is/was the nightmare before es5 (which is much much much more defensive against anti-patterns).

I love redis, so I certainly don't have issue with code written in C, heh. Just code written in C by junior developers :P

Re: Streams: a new general purpose data structure in Redis

#110

Post by erulabs »

takeda wrote: post #91
erulabs wrote: post #66

Earlier quoted context omitted.

Kafka has very poor tooling in my experience (a folder full of fairly buggy bash scripts...), and due to ZooKeeper requires a lot of operational care. For example, it's extremely easy to destroy a Kafka cluster by bringing a new, empty ZK server online with newer but incorrect data in its volume. ZK will happily trash the entire cluster thinking it has new instructions. So network isolation is key, which, while obvio…

> For example, it's extremely easy to destroy a Kafka cluster by bringing a new, empty ZK server online with newer but incorrect data in its volume. ZK will happily trash the entire cluster thinking it has new instructions. How does that happen? I mean a new, empty ZK server with never data than the rest of the cluster? Also, please note that ZK is not meant to be a database, but a coordination service, it's guarante…

Exactly right - in my case the situation was another team accidentally bringing a new ZK node with "bad" but "new" data online. Had there been network isolation, no issues. Had there been static cluster identifiers, also no issues. It was a messy environment, and it should have been prevented by operational diligence, but my point is redis is "harder to mess up". As on on-call engineer, I'll always go with simpler, foolproof tools. Another qibble is how gnarly the client-side driver for Kafka is...

I don't hate Kafka, I just don't like ZK and find redis has better tooling and a better track record at my shops :)

Post reply on HN