Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streams: a new general purpose data structure in Redis

antirez.com

Post reply on HN
Print view View original

121–130 of 154 posts

Re: Streams: a new general purpose data structure in Redis

#121

Post by kochthesecond »

threeseed wrote: post #113
jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

> This is how actual quality software that will stand the test of time gets designed and made .. No not really. It's called waterfall and we had it for decades. And it was largely abandoned in favour of agile development because despite architects spending months designing something there was always something that was left out. Or the scope changed during those months. Or millions of other things that change whilst y…

There is a difference between building products and tools for building products.

Re: Streams: a new general purpose data structure in Redis

#122

Post by rakoo »

lsiebert wrote: post #98
rakoo wrote: post #84

Earlier quoted context omitted.

Yes exactly, you want to ask what is newer than x, where x is the last event you're aware of, but you don't really care about the date and time in that case. If you just store the last id given by redis Streams naively then you don't even care that they're timestamps; at that point my question is, why even bother with the distinction. Just ask for everything after x and be done with it.

Redis also has TIME to get the current server time with milliseconds and the unix time stamp. I'm reasonably sure that's what's being used to get the first part of the ID anyway.

Ah, I didn't know that. Although the post says that the timestamp of an id might also be the timestamp of the last message, since the clock can go backwards, so in the worst case a client might get some duplicate messages.

Re: Streams: a new general purpose data structure in Redis

#123

Post by antirez »

jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

That's an excellent point. I try to do this always, I think that waiting time and thinking more about problems has huge benefits in the final solution. But... when there is a community waiting for new features sometimes this attitude is seen as losing time, and is very hard to explain, so it is important to keep people aware of the design process. Also because via communication it is possible to receive high quality feedbacks. Btw the "slow code" term is great :-)

Re: Streams: a new general purpose data structure in Redis

#124

Post by archit3cture »

femto113 wrote: post #14

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…

In the XADD example, you have

XADD mystream * sensor-id 1234 temperature 10.5

to let the server choose an id.

It could also be interesting to have

XADD mystream 1506872463535.* sensor-id 1234 temperature 10.5

to be able to add an element in a specific msec bucket

Re: Streams: a new general purpose data structure in Redis

#125

Post by styfle »

jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

> And always based on understanding the domain and the existing software you are building upon

This is good. My version of "slow code" is taking note of requests from users and storing in my long term memory to stew over.

1. User A says "It would be nice to have feature X"

2. I ask for more info, what problem are they trying to solve

3. Repeat steps 1 and 2 until user's stop saying "nice to have" and start saying "must have". The stew is now ready and development can begin now that I have thought about a solution that can address everyone's problems...just in time before users get angry :)

Re: Streams: a new general purpose data structure in Redis

#126

Post by mijamo »

anyfoo wrote: post #107
ihattendorf wrote: post #31

Earlier quoted context omitted.

What about regions that treat the comma as a decimal separator? I agree with the other commenter that this is no different than using periods in IPv4 addresses.

As a member of such a region, I suspect that those regions a well aware that the prevalent notation throughout programming uses '.' as a decimal separator. IPv4 addresses usually have 4 components and do not have an inherently fractional unit as their first component.

Disagree. '.' is used for version numbers everywhere, even when it's only 2 components (ex: Wordpress versioning, Django etc.) and it seems pretty clear that 1.11 > 1.2 there.

Once the specification is stated clearly as it is right now it is not a problem.

Another symbol could be used (for instance #) but I really don't see a need for that.

Re: Streams: a new general purpose data structure in Redis

#127

Post by IanCal »

ihattendorf wrote: post #31
femto113 wrote: post #14

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…

What about regions that treat the comma as a decimal separator? I agree with the other commenter that this is no different than using periods in IPv4 addresses.

> What about regions that treat the comma as a decimal separator?

A good reason not to change it from a period to a comma, not really relevant to whether to change it from a period to something else.

> I agree with the other commenter that this is no different than using periods in IPv4 addresses.

IP addresses have no useful concept of "before" or "after", whereas these do.

Re: Streams: a new general purpose data structure in Redis

#128

Post by denkmoon »

baileymiller182 wrote: post #89
luhn wrote: post #19

I'm very excited about this. I've been eying HTTP EventSource for a while now, but there hasn't been a good solution for the backend broker. Kafka is overkill and Amazon Kinesis' pricing isn't viable if you have lots of topics. This fills the need perfectly and Redis is already part of my stack.

Check out nchan.io, an nginx module that does everything you need to connected EventSource to redis backed pub/sub.

+1, good to see other developers out there using EventSource and nchan. We've been happily using both in production for almost a year now.

Re: Streams: a new general purpose data structure in Redis

#130

Post by eternalban »

Gigablah wrote: post #65

Surely any discussion about the concept of logs and mention of Kafka should have a reference to this excellent article by Jay Kreps on LinkedIn: https://engineering.linkedin.com/distributed-systems/log-wha...

It [OP] was a bit cringe worthy, frankly. Obviously the Kafka papers about 'reconsidering the log' and 'unified view' and all that have been out there for years now.

The Kreps article is quite excellent and well worth the read.

Post reply on HN