Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis streams as a pure data structure

antirez.com

Post reply on HN
Print view View original

21–30 of 56 posts

Re: Redis streams as a pure data structure

#21

Post by apeace »

I get that the tennis match use-case is meant to be trivial and an example, but I don't buy it.

> Before Streams we needed to create a sorted set scored by time: the sorted set element would be the ID of the match, living in a different key as a Hash value.

I think the sorted set would be a much better choice, because then you could still insert items in the past, like when that admin remembers there was a tennis match last week he never recorded. Same goes for modifying past values, or deleting values. These operations are trivial using a sorted set & hash, not so using streams.

I'm excited for streams and I'm glad Antirez is taking time to blog and evangelize, but this article didn't convince me there's a compelling use-case for streams aside from the Kafka-like use-case.

Re: Redis streams as a pure data structure

#22

Post by midnightclubbed »

antirez wrote: post #8
midnightclubbed wrote: post #6

Earlier quoted context omitted.

Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful. I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great. I looked for an embedded Redis fork and came up blank, do you have links? I found…

Sorry I don't have links since I did not track such forks in the past. However I've a question: for your use case, isn't it an option to have a library that looks like Redis from the POV of the API, but actually stores objects in memory as data structures native to your programming language? This way the API looks like a mental proxy for the DSL to access Redis and the time complexity you expect from given operations…

That's an option but I would rather not re-invent the wheel unless necessary!

The current use of sqlite is to allow our scripted code (lua and actionscript) to make queries of the exposed data without having to write C++ code for every possible query and data object type (and implement new ones on demand).

Redis might not be the correct thing for this exact use case (some of the queries are more complex than a simple key or range look-up) but I may be prepared to take those limitations in exchange for a substantial speed and/or memory use improvement.

Re: Redis streams as a pure data structure

#23

Post by turtlebits »

siscia wrote: post #20
smush wrote: post #13

Earlier quoted context omitted.

I'm pleased you took the time to send this little targeted advert my way. I will be glad to check out that repo.

Thanks! Any feedback is welcome!

FYI the SSL cert has expired on whoever is hosting your download link (plasso.com)

Re: Redis streams as a pure data structure

#24

Post by antirez »

skybrian wrote: post #18

This seems pretty simple when events are logged as they happen with little or no latency and you can let the stream set the timestamp. I wonder, though, about the case where events may be buffered, perhaps due to an unreliable network? The time that the event occurred might be significantly earlier than the time it's inserted, and furthermore events are arriving out of order. It seems like things get much more compli…

Two solutions: 1. add a timestamp as a field, and just use the ID, but in that case range queries are going to be a problem. 2. exactly because of what you stated, XADD will soon have a special argument to say: I'm going to insert an element in the middle: this is the time in milliseconds (find for me the counter part if I did not specify one). Could be confusing for streaming, but as a data structure to insert in the middle is spot-on and there is nothing preventing that.

Re: Redis streams as a pure data structure

#25

Post by antirez »

apeace wrote: post #21

I get that the tennis match use-case is meant to be trivial and an example, but I don't buy it. > Before Streams we needed to create a sorted set scored by time: the sorted set element would be the ID of the match, living in a different key as a Hash value. I think the sorted set would be a much better choice, because then you could still insert items in the past, like when that admin remembers there was a tennis mat…

We are going to have an option to XADD to insert elements in the middle. I commented more extensively about it in another reply, so inserting out of order later will be possible. However note that the pattern still works when you use a time as a field, you don't need range queries, but just want single-item identifiers. However the XADD option to insert out of order is really a thing that will hit Redis ASAP.

Re: Redis streams as a pure data structure

#26

Post by antirez »

_pgmf wrote: post #17

Streams are kinda cool but they have a distinctly different feel than the other data-types in Redis. They've got this invisible statefulness. Last ids, consumer group state, etc. I've tried implementing a couple little things with streams, and it's not necessary to use the consumer group stuff or whatever of course. I wonder why streams weren't made using the modules API, though? They seem just weird/different enough…

Pure means that when you don't use consumer groups, there is no hidden state at all, and they are just a boring data structure like everything else in Redis. Only if you use the messaging part they have state, but this is an accessory part like a shell on top of what is otherwise exactly a vanilla data structure.

Re: Redis streams as a pure data structure

#27

Post by midnightclubbed »

antirez wrote: post #8
midnightclubbed wrote: post #6

Earlier quoted context omitted.

Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful. I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great. I looked for an embedded Redis fork and came up blank, do you have links? I found…

Sorry I don't have links since I did not track such forks in the past. However I've a question: for your use case, isn't it an option to have a library that looks like Redis from the POV of the API, but actually stores objects in memory as data structures native to your programming language? This way the API looks like a mental proxy for the DSL to access Redis and the time complexity you expect from given operations…

In my case the target application is video games (console and PC) so we performance and/or memory usage critical.

I've used Redis very successfully on the backend, so maybe I'm just trying to find some reason, any reason, to play with it in player facing code!

Re: Redis streams as a pure data structure

#28

Post by kornish »

misterman0 wrote: post #12
antirez wrote: post #5

Earlier quoted context omitted.

There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons. 1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea. 2. As a variation of "1", it's good to have different modules as different processes,…

"basically no gain" My payload are lists of int64's. I need to do set operations on those lists before sending the result over the wire. If you advise against embedding redis, can I instead embed my logic in redis? As a filter of sorts?

You can run Lua natively in Redis.

Re: Redis streams as a pure data structure

#29

Post by siscia »

turtlebits wrote: post #23
siscia wrote: post #20

Earlier quoted context omitted.

Thanks! Any feedback is welcome!

FYI the SSL cert has expired on whoever is hosting your download link (plasso.com)

Thanks! Indeed you should not have see that link! Can you point me where did you clicked?

The correct link is the following now: https://payhip.com/RediSQL

Passo got acquired and shutdown...

Post reply on HN