EventReduce: An algorithm to optimize database queries that run multiple times

Post by eventreduce » Thu, Apr 16, 2020, 3:35 PM UTC

twic wrote: post #50

ComodoHacker wrote: post #37

Earlier quoted context omitted.

And the main cost of this (questionable IMO) benefit is losing consistency, which is losing any change to DB not coming from the calling app. You haven't mentioned this cost anywhere.

> For the different implementations in common browser databases, we can observe an up to 12 times faster displaying of new query results after a write occurred. Is this intended to be an optimisation on top of localStorage and so on? If so, at least you don't have to worry about multiple writers.

Post by eventreduce » Thu, Apr 16, 2020, 3:42 PM UTC

DarkWiiPlayer wrote: post #11

eventreduce wrote: post #5

Earlier quoted context omitted.

1. No I do not have a paper. I thought a lot about publishing a paper first but then decided against it, because I think that good code and tests and demos are more valuable. 2. EventReduce is mostly useful for realtime applications. I myself use it in a NoSQL database (RxDB). There you stream data and events and a single document write is the most atomic 'transaction' you can do. If you need transactional serial wri…

It would probably be a good idea to write a paper at some point; it's simply easier to read a document explaining the algorithm with some pseudocode than to dig through an actual codebase with all the messy language-details in between the parts that actually matter.

Post by eventreduce » Thu, Apr 16, 2020, 4:11 PM UTC

ComodoHacker wrote: post #64

eventreduce wrote: post #60

Earlier quoted context omitted.

You have to provide the events by yourself. See EventReduce as a simple function that can do oldResults+Event=newResults. And yes, you should always do testings before you use open source stuff. There is no warranty use it on your own risk.

OK, so you don't "tunnel writes through" EventReduce, you "tee" them to EventReduce. Anyway, to maintain consistency, you have to limit yourself to one process of your app. No sharding, load-balancing etc. This is significant limitation, and it's not obvious. I encourage you to mention it in README.md.

Post by danpalmer » Thu, Apr 16, 2020, 4:19 PM UTC

eventreduce wrote: post #51

cntlzw wrote: post #46

"EventReduce can be used with relational databases but not on relational queries that run over multiple tables/collections." Forgive my ignorance, but that is the whole point of working with a relational database. If cannot use JOINS then this solves only a very limited use case.

The biggest usecase for EventReduce is realtime applications. Most technologies for these like Firebase, AWS AppSync etc. work on non-relational data. If you want to use EventReduce with relational queries, you have to make them non-relational before, for example by using materialized views. If you do not want to do that, you should not use this algorithm in its current featureset.

Post by truth_seeker » Thu, Apr 16, 2020, 4:51 PM UTC

eventreduce wrote: post #59

truth_seeker wrote: post #56

Earlier quoted context omitted.

>> There is a big difference between a change-stream and a realtime query. For example mongodbs cursor-stream is a good way to observe the events that happen to a specific collection or documents that match some criteria. If you want the realtime-results of a query that has sorting, skip limit etc. than it is really hard to warp the changestream into this. Have you actually tried it from mongo shell or any mongodb cl…

Yes I used it. I actually know it really well. I also did performance comparisons with mongodb and mongodbs change stream and cursors. What I posted here is just an algorithm. You could now compare it to mongodb (a product) and say it is a "more flexible solution" but I do not see the point in directly comparing it simply based on the documentation of both.