Redis streams as a pure data structure
antirez.com
Redis streams as a pure data structure
1–10 of 56 posts
Re: Redis streams as a pure data structure
#2I wish we could standardize on using Redis as general interprocess transactional memory. I could drop 95% of our application code for our Embedded Linux platform by using stock Redis and stock SQLite, but of course there are political obstacles.
Re: Redis streams as a pure data structure
#3I wish we could standardize on using Redis as general interprocess transactional memory. I could drop 95% of our application code for our Embedded Linux platform by using stock Redis and stock SQLite, but of course there are political obstacles.
Is this embedded in the same process, or just within the same unit?
Aside: would an embeddable redis be a useful thing for apps and other isolated devices?
Re: Redis streams as a pure data structure
#4I wish we could standardize on using Redis as general interprocess transactional memory. I could drop 95% of our application code for our Embedded Linux platform by using stock Redis and stock SQLite, but of course there are political obstacles.
I hardly ever comment, but this is a really cool idea. Could you elaborate a little more? (On technical aspects, not political obstacles.)
Re: Redis streams as a pure data structure
#5I wish we could standardize on using Redis as general interprocess transactional memory. I could drop 95% of our application code for our Embedded Linux platform by using stock Redis and stock SQLite, but of course there are political obstacles.
Is this embedded in the same process, or just within the same unit? Aside: would an embeddable redis be a useful thing for apps and other isolated devices?
There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons.
1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea.
2. As a variation of "1", it's good to have different modules as different processes, and Redis works as a glue (message bus) in that case. So again, all should talk to Redis via a unix socket or alike.
3. Latency is usually very acceptable even for the most demanding applications: when it is not, a common pattern to solve such problem is to write to a buffer from within the embedded process, that a different thread moves to Redis. Anyway if you have Redis latencies of any kind, you don't want to block your embedded app main thread.
4. Redis persistence is not compatible with that approach.
5. Many tried such projects (embedded Redis forks or reimplementations) and nobody cared. There must be a reason.
Re: Redis streams as a pure data structure
#6Earlier quoted context omitted.
Is this embedded in the same process, or just within the same unit? Aside: would an embeddable redis be a useful thing for apps and other isolated devices?
There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons. 1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea. 2. As a variation of "1", it's good to have different modules as different processes,…
Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful.
I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great.
I looked for an embedded Redis fork and came up blank, do you have links? I found Vedis, but I would rather have something built off of the Redis code than a re-implementation.
Re: Redis streams as a pure data structure
#7I wish we could standardize on using Redis as general interprocess transactional memory. I could drop 95% of our application code for our Embedded Linux platform by using stock Redis and stock SQLite, but of course there are political obstacles.
I hardly ever comment, but this is a really cool idea. Could you elaborate a little more? (On technical aspects, not political obstacles.)
Complete conjecture, I am not the GP.
Hydrating/deserializing data from Sqlite into types/objects and doing whatever goodness those need, then using Redis to make "updating the database" super fast (in memory after all) and let Redis write it back to Sqlite as there is IO/time/lull in traffic.
Kinda like how Epic Cache does its transaction journal flushing every X minutes?
Re: Redis streams as a pure data structure
#8Earlier quoted context omitted.
There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons. 1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea. 2. As a variation of "1", it's good to have different modules as different processes,…
Having an in-memory datastore that is compact and supports fast queries and flexible data types is very useful. I use sqlite for this purpose, essentially as an in-memory cache of data populated from disk and incoming server packets. Having redis as an option to replace mysql (or at least to compare memory use and speed) would be great. I looked for an embedded Redis fork and came up blank, do you have links? I found…
Sorry I don't have links since I did not track such forks in the past. However I've a question: for your use case, isn't it an option to have a library that looks like Redis from the POV of the API, but actually stores objects in memory as data structures native to your programming language? This way the API looks like a mental proxy for the DSL to access Redis and the time complexity you expect from given operations, but you are just writing to local objects.
Re: Redis streams as a pure data structure
#9Earlier quoted context omitted.
Is this embedded in the same process, or just within the same unit? Aside: would an embeddable redis be a useful thing for apps and other isolated devices?
There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons. 1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea. 2. As a variation of "1", it's good to have different modules as different processes,…
Embedded is a very huge field. That can be everything from aviation, military and medial up to some toys.
For some of those using something as Redis as the existing service might be interesting, for others it will be a no-go.
I worked on automative infotainment system in the past, and throwing Redis on an embedded Linux system there would have been fine if it would have fulfilled a particular task in a good fashion. I think I even proposed it once for something.
Re: Redis streams as a pure data structure
#10Earlier quoted context omitted.
I hardly ever comment, but this is a really cool idea. Could you elaborate a little more? (On technical aspects, not political obstacles.)
Complete conjecture, I am not the GP. Hydrating/deserializing data from Sqlite into types/objects and doing whatever goodness those need, then using Redis to make "updating the database" super fast (in memory after all) and let Redis write it back to Sqlite as there is IO/time/lull in traffic. Kinda like how Epic Cache does its transaction journal flushing every X minutes?
You could have a look into RediSLQ (RediSLQ.com) which is a redis module that embed SQLite, giving crazy fast performance.
It gives you a lot of interesting concepts like "lightweight databases" or push queries result into streams.
Here the GitHub repo: https://github.com/RedBeardLab/rediSQL
Full disclaimer, I am the author.