Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis streams as a pure data structure

antirez.com

Post reply on HN
Print view View original

11–20 of 56 posts

Re: Redis streams as a pure data structure

#11

Post by siscia »

midnightclubbed wrote: post #6
antirez wrote: post #5

Earlier quoted context omitted.

There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons. 1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea. 2. As a variation of "1", it's good to have different modules as different processes,…

Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful. I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great. I looked for an embedded Redis fork and came up blank, do you have links? I found…

When I get started to build RediSLQ I wanted an interprocess, fast, data store that supported SQL manipulation.

It may be useful to you as well: RediSLQ.com

Or on GitHub: https://github.com/RedBeardLab/rediSQL

Full disclaimer: I am the author

Re: Redis streams as a pure data structure

#12

Post by misterman0 »

antirez wrote: post #5
cordite wrote: post #3

Earlier quoted context omitted.

Is this embedded in the same process, or just within the same unit? Aside: would an embeddable redis be a useful thing for apps and other isolated devices?

There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons. 1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea. 2. As a variation of "1", it's good to have different modules as different processes,…

"basically no gain"

My payload are lists of int64's. I need to do set operations on those lists before sending the result over the wire. If you advise against embedding redis, can I instead embed my logic in redis? As a filter of sorts?

Re: Redis streams as a pure data structure

#13

Post by smush »

siscia wrote: post #10
smush wrote: post #7

Earlier quoted context omitted.

Complete conjecture, I am not the GP. Hydrating/deserializing data from Sqlite into types/objects and doing whatever goodness those need, then using Redis to make "updating the database" super fast (in memory after all) and let Redis write it back to Sqlite as there is IO/time/lull in traffic. Kinda like how Epic Cache does its transaction journal flushing every X minutes?

You could have a look into RediSLQ (RediSLQ.com) which is a redis module that embed SQLite, giving crazy fast performance. It gives you a lot of interesting concepts like "lightweight databases" or push queries result into streams. Here the GitHub repo: https://github.com/RedBeardLab/rediSQL Full disclaimer, I am the author.

I'm pleased you took the time to send this little targeted advert my way. I will be glad to check out that repo.

Re: Redis streams as a pure data structure

#14

Post by svnpenn »

siscia wrote: post #10
smush wrote: post #7

Earlier quoted context omitted.

Complete conjecture, I am not the GP. Hydrating/deserializing data from Sqlite into types/objects and doing whatever goodness those need, then using Redis to make "updating the database" super fast (in memory after all) and let Redis write it back to Sqlite as there is IO/time/lull in traffic. Kinda like how Epic Cache does its transaction journal flushing every X minutes?

You could have a look into RediSLQ (RediSLQ.com) which is a redis module that embed SQLite, giving crazy fast performance. It gives you a lot of interesting concepts like "lightweight databases" or push queries result into streams. Here the GitHub repo: https://github.com/RedBeardLab/rediSQL Full disclaimer, I am the author.

> RediSLQ

> rediSQL

why different spelling

Re: Redis streams as a pure data structure

#15

Post by e12e »

antirez wrote: post #8
midnightclubbed wrote: post #6

Earlier quoted context omitted.

Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful. I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great. I looked for an embedded Redis fork and came up blank, do you have links? I found…

Sorry I don't have links since I did not track such forks in the past. However I've a question: for your use case, isn't it an option to have a library that looks like Redis from the POV of the API, but actually stores objects in memory as data structures native to your programming language? This way the API looks like a mental proxy for the DSL to access Redis and the time complexity you expect from given operations…

It sounds to me like there might be some area where sqlite is "to much" but lightningdb/berkleydb/toky cabinet is "too little".

I would be surprised if "actual Redis" was ever the right answer to "sqlite is too much".

But I do wonder if there are some lessons to take from Redis api and wrap something like lmdb/bdb etc.

I'm not familiar enough with Redis to know when/if this would make sense over just using sqlite, though.

Re: Redis streams as a pure data structure

#16

Post by e12e »

siscia wrote: post #10
smush wrote: post #7

Earlier quoted context omitted.

Complete conjecture, I am not the GP. Hydrating/deserializing data from Sqlite into types/objects and doing whatever goodness those need, then using Redis to make "updating the database" super fast (in memory after all) and let Redis write it back to Sqlite as there is IO/time/lull in traffic. Kinda like how Epic Cache does its transaction journal flushing every X minutes?

You could have a look into RediSLQ (RediSLQ.com) which is a redis module that embed SQLite, giving crazy fast performance. It gives you a lot of interesting concepts like "lightweight databases" or push queries result into streams. Here the GitHub repo: https://github.com/RedBeardLab/rediSQL Full disclaimer, I am the author.

It looks like an interesting project - but I'm not sure I understand how it's better than a ram backed sqlite instance. It forces/let's you use the Redis protocol to connect rather than embedding?

Re: Redis streams as a pure data structure

#17

Post by _pgmf »

Streams are kinda cool but they have a distinctly different feel than the other data-types in Redis. They've got this invisible statefulness. Last ids, consumer group state, etc. I've tried implementing a couple little things with streams, and it's not necessary to use the consumer group stuff or whatever of course. I wonder why streams weren't made using the modules API, though? They seem just weird/different enough to warrant exclusion from the core data types, in my thinking. Anyways, just reading the title referring to streams as "pure" made me go wtf? Because there's a lot of hidden state in there.

Re: Redis streams as a pure data structure

#18

Post by skybrian »

This seems pretty simple when events are logged as they happen with little or no latency and you can let the stream set the timestamp. I wonder, though, about the case where events may be buffered, perhaps due to an unreliable network? The time that the event occurred might be significantly earlier than the time it's inserted, and furthermore events are arriving out of order. It seems like things get much more complicated?

Let's say tennis games are recorded on a piece of paper and entered into the computer later. What is different?

Re: Redis streams as a pure data structure

#19

Post by siscia »

e12e wrote: post #16
siscia wrote: post #10

Earlier quoted context omitted.

You could have a look into RediSLQ (RediSLQ.com) which is a redis module that embed SQLite, giving crazy fast performance. It gives you a lot of interesting concepts like "lightweight databases" or push queries result into streams. Here the GitHub repo: https://github.com/RedBeardLab/rediSQL Full disclaimer, I am the author.

It looks like an interesting project - but I'm not sure I understand how it's better than a ram backed sqlite instance. It forces/let's you use the Redis protocol to connect rather than embedding?

It allow you to access the same dataset from different processes (possible also with SQLite) and machines.

Re: Redis streams as a pure data structure

#20

Post by siscia »

smush wrote: post #13
siscia wrote: post #10

Earlier quoted context omitted.

You could have a look into RediSLQ (RediSLQ.com) which is a redis module that embed SQLite, giving crazy fast performance. It gives you a lot of interesting concepts like "lightweight databases" or push queries result into streams. Here the GitHub repo: https://github.com/RedBeardLab/rediSQL Full disclaimer, I am the author.

I'm pleased you took the time to send this little targeted advert my way. I will be glad to check out that repo.

Thanks! Any feedback is welcome!
Post reply on HN