Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streams: a new general purpose data structure in Redis

antirez.com

Post reply on HN
Print view View original

141–150 of 154 posts

Re: Streams: a new general purpose data structure in Redis

#141

Post by irrational »

threeseed wrote: post #113
jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

> This is how actual quality software that will stand the test of time gets designed and made .. No not really. It's called waterfall and we had it for decades. And it was largely abandoned in favour of agile development because despite architects spending months designing something there was always something that was left out. Or the scope changed during those months. Or millions of other things that change whilst y…

But waterfall or agile implies code is being written. My take on this (and experience) is that sometimes you just takes weeks, months, even years to just make notes in a notebook, ponder and think upon the problem, jot down more notes, look around at other peoples code, do research to see if anyone else has done work in this area, do more pondering and thinking and note taking, talk it over with other people, etc. At some point it all gels together (when that happens it is almost magical and such an amazing moment of clarity) and then you are ready to use some sort of development process to actually write the code. Though by that point writing the code is extremely easy since you have such clear insight into what needs to be written. I love when that happens.

Re: Streams: a new general purpose data structure in Redis

#142

Post by omni »

chicagobuss wrote: post #32
agacera wrote: post #17

Earlier quoted context omitted.

I can think in some circumtances: 1 - You already have a Redis infrastructure and don't wanna or don't have resources to deploy a full Kafka infrastructure (3 kafka brokers + 3 zookeeper nodes) 2 - Kafka clients are not available (or are poorly available) for every programming language. Redis has a simpler protocol, so it has more/and better clients available and even if you use an exotic language, it is easy to writ…

re: client support - I dunno, this seems like a pretty comprehensive list to me? I mean, there's even a rust client: https://cwiki.apache.org/confluence/display/KAFKA/Clients

As someone who has been burned by non-Java Kafka drivers, beware the perception of ecosystem support here. The Kafka design pushes a huge amount of complexity onto the client, and in our experience only the Java client deals with this complexity well. We started out using Python clients but eventually moved to Confluent's REST API (wrapping the Java driver) because we had so many problems with it.

Re: Streams: a new general purpose data structure in Redis

#143

Post by dvt »

manigandham wrote: post #135
dvt wrote: post #99

Earlier quoted context omitted.

Of course it does: thought/discussion take time.

It seems people are confused. Of course everything takes physical time, but slowly thinking about something isn't any objective sign of better outcomes.

I think the salient point is that thinking takes more time than not thinking. At least in my experience, the policy is generally to push features, not to think (slowly or otherwise).

Re: Streams: a new general purpose data structure in Redis

#144

Post by takeda »

erulabs wrote: post #110
takeda wrote: post #91

Earlier quoted context omitted.

> For example, it's extremely easy to destroy a Kafka cluster by bringing a new, empty ZK server online with newer but incorrect data in its volume. ZK will happily trash the entire cluster thinking it has new instructions. How does that happen? I mean a new, empty ZK server with never data than the rest of the cluster? Also, please note that ZK is not meant to be a database, but a coordination service, it's guarante…

Exactly right - in my case the situation was another team accidentally bringing a new ZK node with "bad" but "new" data online. Had there been network isolation, no issues. Had there been static cluster identifiers, also no issues. It was a messy environment, and it should have been prevented by operational diligence, but my point is redis is "harder to mess up". As on on-call engineer, I'll always go with simpler, f…

In order to connect a ZK host to the cluster its IP needs to be included in configuration of all the nodes.

It's hard to accidentally add node to a cluster. A person who can "accidentally" add a ZK node has enough permission to do a lot of more devastating things accidentally.

Re: Streams: a new general purpose data structure in Redis

#146

Post by jrochkind1 »

manigandham wrote: post #135
dvt wrote: post #99

Earlier quoted context omitted.

Of course it does: thought/discussion take time.

It seems people are confused. Of course everything takes physical time, but slowly thinking about something isn't any objective sign of better outcomes.

Everyone works differently, but for me thinking in a leisurely way does produce better outcomes. It's less that my actual thinking is somehow "slower", than that thinking about something and then letting it rest and come back to it, I have better conclusions than if I rushed to judgement.

I guess the phrase 'rush to judgement' is instructive. We're not just thinking, we're deciding of course.

If you can find out about a problem or issue or desire and very quickly come up with an architectural solution that will stand the test of time for years (at least without backwards-incompatibilities, which is what a redis requires), more power to you, but in my experience and observation most will not.

But I get it, you just don't agree with my basic suggestion, from your experience. That's fine. It's not really an argument about the speed of one's thinking. Nobody is "confused".

Re: Streams: a new general purpose data structure in Redis

#147

Post by femto113 »

femto113 wrote: post #102
antirez wrote: post #82

Earlier quoted context omitted.

Not sure... : looks ok actually, even _ or - could make some sense. The # is a bit too heavy on the eyes :-)

I'll probably eventually hate myself for even bringing this up but I can't help but notice the similarity between this ID structure and Version 1 (aka timestamp) UUIDs. While I wouldn't go as far as recommending that you fully adopt that form, it might be worth considering if you could make these IDs compatible with UUIDs by defining a canonical transform. The critical differences are: - UUIDs use a different epoch (…

Thinking about this some more I realized you could also encode sequence into the low order bits of the timestamp, and rereading the RFC showed it actually makes this recommendation[1]. There are 10000 100-nanosecond periods per millisecond which gives about 13 more bits. Between that and the 13-15 bits available in the clock sequence you've got 26+ bits of sequence or ~67MM values per millisecond.

Since 64 bits is overkill for milliseconds (45 bits covers the next 1000 years or so) I was thinking you could put 2 bytes of the node id in the high order bytes there (perhaps could call this the "clock id"?) and the remaining 4 bytes of the node id could go in the high order bytes of the sequence, which would still leave 32 bits for actual sequence values (but we should only use 26 or so). This means we'd get a translation roughly as follows (numbering bytes and bits from high to low significance):

   Redis                                   Version 1 UUID
      Timestamp
        Byte 0-1  "clock id"               Bytes 4&5 of node id
        Byte 2-7  millis since 1 Jan 1970  * 10000 => ~45 high order timestamp bits
      Sequence
        Byte 0-3  "node id"                Bytes 0-3 of node id
        Byte 4-7
           6 bits wasted space             ignored
          26 bits actual sequence value
            13 high order bits             => clock sequence
            13 low order bits              => low order timestamp bits
Another implication of this scheme is that if redis has access to a clock that offers higher than millisecond resolution it could store everything more precise than millisecond into the sequence portion of the id.

On a side note it seems that the clock sequence in the UUID is intended to be reset to a random value at start up and every time a clock jump is detected rather than just incremented. Redis could do something similar by incrementing some of the 13 high-order bits of the sequence every time a clock jump is detected (and/or if the 13 low-order bits overflow)

[1] https://tools.ietf.org/html/rfc4122#section-4.2.1.2

Re: Streams: a new general purpose data structure in Redis

#148

Post by dbattaglia »

ralusek wrote: post #37

I love redis, and this API looks amazingly simple. I'm sure I can think up a good use case for this, but the only problem I have with it is that time-series log data of this nature is increasingly becoming the defacto source of truth in the various models resembling some version or another of event-sourcing. Obviously the general thinking is that event sourced time series data allows you to treat every other data sou…

I would imagine this is best used for the "store a ton of events with some capped maximum size" Kafka use case (ie - realtime analytics, IoT data, etc). I just can't imagine using Redis as your source of truth for an event sourced system, especially without the partitioning and log compacting features that Kafka has. Still this seems like a pretty amazing feature to have in Redis, can't wait to start playing with it.

Re: Streams: a new general purpose data structure in Redis

#149

Post by scott_karana »

freshhawk wrote: post #64
barrkel wrote: post #45

Earlier quoted context omitted.

It's also why software is a pop culture. Sophistication and completeness is seen as complexity and cruft by each successive generation, who start something new and simple. I don't think it's very avoidable. Tech is genuinely getting incrementally better, but it's usually in a sawtooth pattern.

I'll agree that sophistication and completeness is often seen as complexity/cruft but it also always comes with actual cruft as well since improvement is incremental and breaking APIs is annoying. My favorite aspect of this cycle is when some features in the complex software become seen as so useful as to be required and standard, so when the new simpler version is created they have to figure out a novel way of provi…

Do you have any examples of that second case?

Sounds too good to be true... but I'd love to be proven wrong :)

Re: Streams: a new general purpose data structure in Redis

#150

Post by d2clon »

jrochkind1 wrote: post #88

One thing I like about this post is the story of how the feature came to be: Someone who understood redis very well, thinking about the problem over literally years, eventually resulting in a more targetted and goal-driven thinking, and even that "specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I acc…

I feel very reflected in this "slow code" interpretation. Instinctively, I try to refrain my self to implement this new feature that looks important but not urgent. The positive outcome is not only that you have more time to think about the proper approach but also eventually, some times, the feature shows it self not needed any more.
Post reply on HN