Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis streams as a pure data structure

antirez.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Redis streams as a pure data structure

#31

Post by nicpottier »

antirez wrote: post #25
apeace wrote: post #21

I get that the tennis match use-case is meant to be trivial and an example, but I don't buy it. > Before Streams we needed to create a sorted set scored by time: the sorted set element would be the ID of the match, living in a different key as a Hash value. I think the sorted set would be a much better choice, because then you could still insert items in the past, like when that admin remembers there was a tennis mat…

We are going to have an option to XADD to insert elements in the middle. I commented more extensively about it in another reply, so inserting out of order later will be possible. However note that the pattern still works when you use a time as a field, you don't need range queries, but just want single-item identifiers. However the XADD option to insert out of order is really a thing that will hit Redis ASAP.

Excellent to hear this.

We use sorted sets as queues heavily and this would be a necessary thing for us to consider giving streams a go which would indeed be interesting from a memory savings (we sometimes have millions of items in our queues for a short time). Sometimes, say on error conditions, you want to stuff something back at the start of the queue (because the order of processing matters) instead of at the end as one example.. priority being another.

Re: Redis streams as a pure data structure

#32

Post by skrebbel »

Did anyone yet use Redis streams to store actual logs? Like server logs, application logs, etc.

I understand that Elasticsearch is a common place to put logs, also because I assume that searching through logs is a common use case, but I wonder whether Redis has particular benefits for this use case. The data structure seems particularly tailored to it (but not so much to searching I guess).

Re: Redis streams as a pure data structure

#33

Post by atombender »

skrebbel wrote: post #32

Did anyone yet use Redis streams to store actual logs? Like server logs, application logs, etc. I understand that Elasticsearch is a common place to put logs, also because I assume that searching through logs is a common use case, but I wonder whether Redis has particular benefits for this use case. The data structure seems particularly tailored to it (but not so much to searching I guess).

My guess is that this would work fine until the working set size exceeds available memory. Redis (unless something new has happened the last couple of years since I used it) requires that data fit in RAM. So could work well for low-frequency logging like alerts. Not as a general purpose log system.

Re: Redis streams as a pure data structure

#34

Post by rainhacker »

reggieband wrote: post #30

I wonder how this compares to streams in Kafka or Kinesis. One of the main advantages of redis is that I see it used in many cases as a replacement for memcache (just a key/value store for bytes/strings) so it already exists in many infrastructures.

I shared my experience sometime back in another HN thread [1]:

"A key difference I observed was that if a Kafka consumer crashes, a rebalance is triggered by Kafka after which the remaining consumers seamlessly start consuming the messages from the last committed offset of the failed consumer.

Whereas with Redis streams I had to write code in my application to periodically poll and claim unacked messages pending for more than some threshold time."

[1] https://news.ycombinator.com/item?id=19231178

Re: Redis streams as a pure data structure

#35

Post by antoncohen »

skrebbel wrote: post #32

Did anyone yet use Redis streams to store actual logs? Like server logs, application logs, etc. I understand that Elasticsearch is a common place to put logs, also because I assume that searching through logs is a common use case, but I wonder whether Redis has particular benefits for this use case. The data structure seems particularly tailored to it (but not so much to searching I guess).

Log volume can easily exceed reasonable memory sizes. Even a small company can generate TBs of logs each month. Having a single box with TBs of memory wouldn't be desirable.

For logs without full indexing, Loki (https://github.com/grafana/loki) is a recent entry into the space, and it probably a good option to look at. It indexes metadata (labels), so it allows searching by labels but not full text. It is also supposed to be horizontally-scalable, which is probably something you want in a log storage solution.

Re: Redis streams as a pure data structure

#36

Post by [deleted] »

skybrian wrote: post #18

This seems pretty simple when events are logged as they happen with little or no latency and you can let the stream set the timestamp. I wonder, though, about the case where events may be buffered, perhaps due to an unreliable network? The time that the event occurred might be significantly earlier than the time it's inserted, and furthermore events are arriving out of order. It seems like things get much more compli…

[deleted]

Re: Redis streams as a pure data structure

#37

Post by coleifer »

antirez wrote: post #26
_pgmf wrote: post #17

Streams are kinda cool but they have a distinctly different feel than the other data-types in Redis. They've got this invisible statefulness. Last ids, consumer group state, etc. I've tried implementing a couple little things with streams, and it's not necessary to use the consumer group stuff or whatever of course. I wonder why streams weren't made using the modules API, though? They seem just weird/different enough…

Pure means that when you don't use consumer groups, there is no hidden state at all, and they are just a boring data structure like everything else in Redis. Only if you use the messaging part they have state, but this is an accessory part like a shell on top of what is otherwise exactly a vanilla data structure.

Can you read from a stream that doesn't exist yet?

Re: Redis streams as a pure data structure

#38

Post by jinjin2 »

midnightclubbed wrote: post #6
antirez wrote: post #5

Earlier quoted context omitted.

There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons. 1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea. 2. As a variation of "1", it's good to have different modules as different processes,…

Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful. I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great. I looked for an embedded Redis fork and came up blank, do you have links? I found…

I’ve used Realm for this very successfully. It is a bit limited in the number of languages it supports (outside of mobile where it seems to support pretty much everything), but it has really nice support for node.js and .net which is where I have used it.

It is pretty cool to be able to share live interconnected objects between processes with full transactional safety.

Re: Redis streams as a pure data structure

#39

Post by opportune »

rainhacker wrote: post #34
reggieband wrote: post #30

I wonder how this compares to streams in Kafka or Kinesis. One of the main advantages of redis is that I see it used in many cases as a replacement for memcache (just a key/value store for bytes/strings) so it already exists in many infrastructures.

I shared my experience sometime back in another HN thread [1]: "A key difference I observed was that if a Kafka consumer crashes, a rebalance is triggered by Kafka after which the remaining consumers seamlessly start consuming the messages from the last committed offset of the failed consumer. Whereas with Redis streams I had to write code in my application to periodically poll and claim unacked messages pending for…

From my experience, Kafka has the best api for handling read-once, distributed streams. Almost every other streaming solution, like Redis in this case, has a non-ideal or non-existent way to coordinate stream consumers in a way that prevents double-reads. And lots of streaming applications need to ensure read-once (think about what a double read ends up as - maybe a twice-sent message, or a duplicate metric), so I'm not sure why they all struggle so much with just copying kafka's pretty simple consumer api

Re: Redis streams as a pure data structure

#40

Post by 64738 »

siscia wrote: post #11
midnightclubbed wrote: post #6

Earlier quoted context omitted.

Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful. I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great. I looked for an embedded Redis fork and came up blank, do you have links? I found…

When I get started to build RediSLQ I wanted an interprocess, fast, data store that supported SQL manipulation. It may be useful to you as well: RediSLQ.com Or on GitHub: https://github.com/RedBeardLab/rediSQL Full disclaimer: I am the author

Did you mean RediSQL.com? Or is that someone else?
Post reply on HN