Live data from Hacker News

Redis streams as a pure data structure

antirez.com

41–50 of 56 posts

Re: Redis streams as a pure data structure

#41
Just like it's useful to have both SQLite and Postgres available for smaller and larger data projects (and Spatialite and PostGIS for smaller/larger geo-data projects), it could be great to have Redis and Kafka for smaller and larger pipeline projects.

Does anyone have good patterns for joining across entries from two or more Redis streams? This is one of the most interesting aspects of Kafka/Flink/Spark/Storm/etc. Would be useful to be able to develop with streaming joins in Redis playgrounds.

Re: Redis streams as a pure data structure

#42
post #40
post #11

Earlier quoted context omitted.

When I get started to build RediSLQ I wanted an interprocess, fast, data store that supported SQL manipulation. It may be useful to you as well: RediSLQ.com Or on GitHub: https://github.com/RedBeardLab/rediSQL Full disclaimer: I am the author

Did you mean RediSQL.com? Or is that someone else?

Indeed you are right!

Yes https://redisql.com

Re: Redis streams as a pure data structure

#43
post #10
post #7

Earlier quoted context omitted.

Complete conjecture, I am not the GP. Hydrating/deserializing data from Sqlite into types/objects and doing whatever goodness those need, then using Redis to make "updating the database" super fast (in memory after all) and let Redis write it back to Sqlite as there is IO/time/lull in traffic. Kinda like how Epic Cache does its transaction journal flushing every X minutes?

You could have a look into RediSLQ (RediSLQ.com) which is a redis module that embed SQLite, giving crazy fast performance. It gives you a lot of interesting concepts like "lightweight databases" or push queries result into streams. Here the GitHub repo: https://github.com/RedBeardLab/rediSQL Full disclaimer, I am the author.

The correct URL is https://redisql.com

Re: Redis streams as a pure data structure

#44
Streams are great! I've written a small library for Node which attempts to wrap some of the complexity (particularly for handling multiple connections for UNBLOCK calls, etc).

So far I haven't used it outside of hobby projects for webGL games and such, but it's worked brilliantly, and no Kafka required for hobby async-streaming infrastructure!

Hopefully it's useful to someone out there! https://github.com/erulabs/redis-streams-aggregator

Re: Redis streams as a pure data structure

#45
post #20
post #13

Earlier quoted context omitted.

I'm pleased you took the time to send this little targeted advert my way. I will be glad to check out that repo.

Thanks! Any feedback is welcome!

Some silly feedback on typos:

> Carefully tested for correcteness and tuned for performance – never loose a bit.

correcteness -> correctness (slightly ironic :-) ) loose -> lose

> RediSQL inheritanve all the SQLite knobs and capabilities

inheritanve -> inherits

> RediSQL is written in Rust which provides more guarantess agains common bugs

agains -> against

> Only a very minor part of RediSQL is not releases as open source

releases -> released

Just fyi :-) Looks like a really interesting tool :-)

Re: Redis streams as a pure data structure

#46

Earlier quoted context omitted.

I shared my experience sometime back in another HN thread [1]: "A key difference I observed was that if a Kafka consumer crashes, a rebalance is triggered by Kafka after which the remaining consumers seamlessly start consuming the messages from the last committed offset of the failed consumer. Whereas with Redis streams I had to write code in my application to periodically poll and claim unacked messages pending for…

From my experience, Kafka has the best api for handling read-once, distributed streams. Almost every other streaming solution, like Redis in this case, has a non-ideal or non-existent way to coordinate stream consumers in a way that prevents double-reads. And lots of streaming applications need to ensure read-once (think about what a double read ends up as - maybe a twice-sent message, or a duplicate metric), so I'm…

[deleted]

Re: Redis streams as a pure data structure

#47
post #5
post #3

Earlier quoted context omitted.

Is this embedded in the same process, or just within the same unit? Aside: would an embeddable redis be a useful thing for apps and other isolated devices?

There is basically no gain in practical terms in running Redis as an embedded library in embedded contexts, at this point I think I'm able to summarize the key reasons. 1. Embedded systems are often used in environments where you need very resilient software. To crash the DB because there is a bug in your app is usually a bad idea. 2. As a variation of "1", it's good to have different modules as different processes,…

I beg to differ. SQLite is a very popular embedded database. There is inherent simplicity to just reading and writing flat files.

Redis feels like that. It’s a simple data structure server. Now if we could have those datastructurs sync with flatfiles with the same redis API, a lot of applications would become much simpler.

I’m not sure how big of an undertaking it is though.

I’m willing to bet, a fast general datasrtuctures database syncable to flat files would open up many possibilities.

Re: Redis streams as a pure data structure

#48
post #15
post #8

Earlier quoted context omitted.

Sorry I don't have links since I did not track such forks in the past. However I've a question: for your use case, isn't it an option to have a library that looks like Redis from the POV of the API, but actually stores objects in memory as data structures native to your programming language? This way the API looks like a mental proxy for the DSL to access Redis and the time complexity you expect from given operations…

It sounds to me like there might be some area where sqlite is "to much" but lightningdb/berkleydb/toky cabinet is "too little". I would be surprised if "actual Redis" was ever the right answer to "sqlite is too much". But I do wonder if there are some lessons to take from Redis api and wrap something like lmdb/bdb etc. I'm not familiar enough with Redis to know when/if this would make sense over just using sqlite, th…

One limitation of SQLite is that it doesn’t support any kind of “notify me when some other process does X” operation. (If you Google it, you’ll find sqlite3_update_hook, but that only works for updates performed by the same process.) If you want to use SQLite as an event queue, you can have one process writing rows to a table and another process reading them, but you need some external signal to tell the second process “wake up, there’s new stuff in the queue”. Or you can have it poll on a timer, but that’s suboptimal in many different ways.

Which is topical, because watching for updates is a core feature of Redis streams (and Redis already had pub/sub channels before that). For that use case, SQLite is too little, even if your needs are otherwise quite basic.

Unfortunately, this difference in capabilities seems to be partly a result of limitations in the underlying OS APIs. SQLite uses POSIX advisory locks to lock ranges of the database file, but I don’t think there’s any similar API that provides an event or semaphore associated with a given file, instead of a lock. There are plenty of messaging APIs that aren’t associated with an arbitrary file – there are semaphores, message queues, and shared memory objects, in fact two sets of APIs for each of those (SysV and POSIX), plus signals, etc. But those all have their own namespaces, and if the two processes trying to synchronize with each other are in different containers, they might not share those namespaces. There are Unix sockets – those are a decent option, but they require one process to set itself up as the server, which is a bit weird in the SQLite model where all the processes are on an equal footing, and any may quit at any time. They also don’t work over NFS (whereas locking does, at least sometimes). You can try to mmap a regular file and then treat it as shared memory, but that’s not guaranteed to work in all cases, and again doesn’t work over NFS. I suppose you could try to abuse a lock as a semaphore, but that has its limitations…

But it’s not like many people use SQLite over NFS anyway. Whatever the approach, I’d love to see a “SQLite for notifications”. It would probably be a pretty simple library, but with the needed bells and whistles like bindings to higher level languages. If a library like this exists, I’d be very interested to hear, because a while back I searched for one in vain.

Re: Redis streams as a pure data structure

#49

Earlier quoted context omitted.

I shared my experience sometime back in another HN thread [1]: "A key difference I observed was that if a Kafka consumer crashes, a rebalance is triggered by Kafka after which the remaining consumers seamlessly start consuming the messages from the last committed offset of the failed consumer. Whereas with Redis streams I had to write code in my application to periodically poll and claim unacked messages pending for…

From my experience, Kafka has the best api for handling read-once, distributed streams. Almost every other streaming solution, like Redis in this case, has a non-ideal or non-existent way to coordinate stream consumers in a way that prevents double-reads. And lots of streaming applications need to ensure read-once (think about what a double read ends up as - maybe a twice-sent message, or a duplicate metric), so I'm…

How so? Kafka only accepts offsets which are meant for batches of items or even an entire partition. This means a single item not being processed within that batch requires your own code to compensate. It's the weakest of all the messaging models.

Per-message acknowledgement is an advancement. Redis requires manual lookups for unacknowledged items but you can also use Apache Pulsar for a more scalable distributed disk-based system which itself is a solid evolution over Kafka's design.

Also note that "exactly-once" semantics are actually impossible. Messaging systems are either "at-least-once" or "at-most-once". Kafka has some attempts at using transactions to solve this but that's only when using Kafka streams and only ensures read progress, not the processing result.

Re: Redis streams as a pure data structure

#50
post #29

Earlier quoted context omitted.

FYI the SSL cert has expired on whoever is hosting your download link (plasso.com)

Thanks! Indeed you should not have see that link! Can you point me where did you clicked? The correct link is the following now: https://payhip.com/RediSQL Passo got acquired and shutdown...

On mobile there is a "Buy Now" button at the top of the screen that goes to plasso. https://imgur.com/a/XYD9slH
Post reply on HN