Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streams: a new general purpose data structure in Redis

antirez.com

Post reply on HN
Print view View original

51–60 of 154 posts

Re: Streams: a new general purpose data structure in Redis

#51

Post by Posibyte »

hueving wrote: post #48
antirez wrote: post #40

Earlier quoted context omitted.

Hello, the streams have basically the same characteristics as any other Redis data structure, that is, from the POV of a local node, you can configure strong persistence on disk, but on node failures, you have basically different tunable amount of best effort consistency, it means that you cannot guarantee no messages are lost. So basically this means that you can: 1. Use the default asynchronous replication, and liv…

What is the consistency model of redis? It sounds like anything can be lost in redis during normal HA operations even with WAIT pushing to a majority of slaves. Is that right?

Eventually consistent unless you're only using a single node. I believe that Redis itself commits to disk at various checkpoints in time, so if a fail happens, you're really only guaranteed to fail over into a pool of data that's consistent up to the last checkpoint of the node you're moving to.

EDIT: And as antirez said above, you can WAIT to force synchronization to all nodes, so you would be pretty likely to fail over onto a node that has n-1 messages if it didn't sync in time. That still isn't guaranteed however.

Re: Streams: a new general purpose data structure in Redis

#52

Post by sidlls »

hueving wrote: post #50
sidlls wrote: post #9

Earlier quoted context omitted.

In any context its used where the demand (by whatever measure you care to use: bandwidth, throughput, message durability, etc.) doesn't justify it or isn't a good use case of Kafka, for starters. That happens all the time, because every data and infrastructure engineer in the Bay Area wants to put Kafka on his resume.

But that doesn't explain why Kafka has any minimum the output required. Does it have usability issues? A good tool should be able to be used at any scale.

> A good tool should be able to be used at any scale.

I don't necessarily agree that a tool should be used at any scale even if it's technically possible to. Cost (multiple dimensions, including money and engineering effort) factors in.

Re: Streams: a new general purpose data structure in Redis

#53

Post by hakanito »

luhn wrote: post #19

I'm very excited about this. I've been eying HTTP EventSource for a while now, but there hasn't been a good solution for the backend broker. Kafka is overkill and Amazon Kinesis' pricing isn't viable if you have lots of topics. This fills the need perfectly and Redis is already part of my stack.

Just wanted to chip in and say HTTP EventSource has been really nice to work with, we've been using it in production for 1+ year

Re: Streams: a new general purpose data structure in Redis

#55

Post by antirez »

ricardobeat wrote: post #54

Slightly off-topic, but could the blog be adjust a little for mobile reading? #content { max-width: 800px; } // replaces width: 800px seems to do the job, and also behaves better in narrow desktop browser windows.

Thanks! I'll change it tonight.

Re: Streams: a new general purpose data structure in Redis

#56

Post by rakoo »

antirez wrote: post #20
femto113 wrote: post #14

I have a confusion about ID structure/format: The ID is composed of two parts: a millisecond time and a sequence number. The number after the dot is the sequence number, and is used in order to distinguish entries added in the same millisecond. Does this mean for example that 1506872463535.11 comes after 1506872463535.2 (because 11 > 2)? If so that means treating these as decimals (which will be easy to do inadverten…

Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.

Why keep the separation at all ? Are clients expected to be able to query for a given timestamp precisely ? Because then you get all the problems with clock synchronization, especially given that the Streams' clock is monotonic and I'd expect clients' clock to not be

Re: Streams: a new general purpose data structure in Redis

#57

Post by jankotek »

wyc wrote: post #10

Projects tend to gain more and more functionality to match the new workloads they're being used to accomplish, and it must certainly be a difficult decision for project visionaries. Do I listen to my users and implement features that will solve their new woes, but in return accept increased complexity and higher learning barriers? Complexity sucks, but it's even harder to say no to users in pain. I wonder if this ope…

Look at the source code. Redis is very simple, single threaded, primitive networking... I would hardly call it complex even with this extra feature.

Re: Streams: a new general purpose data structure in Redis

#58

Post by rakoo »

luhn wrote: post #19

I'm very excited about this. I've been eying HTTP EventSource for a while now, but there hasn't been a good solution for the backend broker. Kafka is overkill and Amazon Kinesis' pricing isn't viable if you have lots of topics. This fills the need perfectly and Redis is already part of my stack.

I don't know if you've ever looked at it, but CouchDB has had an EventSource endpoint for a long time now (http://docs.couchdb.org/en/2.1.0/api/database/changes.html?h...). CouchDB is extremely easy to install, use and maintain, and there's a number of public providers out there if you don't want to host everything yourself.

As a more generic solution, there's also pushpin (http://pushpin.org/), which is the backend of fanout (https://fanout.io/), so that may also be a nice addition to your stack if you want a more direct redis->clients link

Re: Streams: a new general purpose data structure in Redis

#59

Post by ricardobeat »

Could the difference between `MAXLEN ~ 1000000` and `MAXLEN 100000` be handled internally, by marking the overflowing items as deleted until a whole block can be removed? Looks like this tombstone functionality is already planned, would make the API simpler.

Re: Streams: a new general purpose data structure in Redis

#60

Post by dvirsky »

ryanschneider wrote: post #27
antirez wrote: post #20

Earlier quoted context omitted.

Yes actually maybe it's a good idea to change the point with something else. Thanks for the hint.

Supporting `MAXSIZE` as well as `MAXLEN` on `XADD` would also handle a nice Kafka feature (the ability to define your log size in either number of messages or size on disk). Something like: `XADD MAXSIZE ~ 2147484000 * foo bar` to cap the stream at 2GB + 1 node.

And if ids are timestamps, maybe we can define it as MAXTIME as well.
Post reply on HN