Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

EventReduce: An algorithm to optimize database queries that run multiple times

github.com

Post reply on HN
Print view View original

51–60 of 87 posts

Re: EventReduce: An algorithm to optimize database queries that run multiple times

#51

Post by eventreduce »

cntlzw wrote: post #46

"EventReduce can be used with relational databases but not on relational queries that run over multiple tables/collections." Forgive my ignorance, but that is the whole point of working with a relational database. If cannot use JOINS then this solves only a very limited use case.

The biggest usecase for EventReduce is realtime applications. Most technologies for these like Firebase, AWS AppSync etc. work on non-relational data. If you want to use EventReduce with relational queries, you have to make them non-relational before, for example by using materialized views. If you do not want to do that, you should not use this algorithm in its current featureset.

Re: EventReduce: An algorithm to optimize database queries that run multiple times

#53

Post by eventreduce »

lovasoa wrote: post #49
cntlzw wrote: post #46

"EventReduce can be used with relational databases but not on relational queries that run over multiple tables/collections." Forgive my ignorance, but that is the whole point of working with a relational database. If cannot use JOINS then this solves only a very limited use case.

Noria [1] is a research database that solves the same problem while still supporting all relational database queries. [1] https://github.com/mit-pdos/noria

I think it is dangerous to propose a database product as a solution to the limitation of a simple algorithm.

Re: EventReduce: An algorithm to optimize database queries that run multiple times

#54

Post by cntlzw »

eventreduce wrote: post #51
cntlzw wrote: post #46

"EventReduce can be used with relational databases but not on relational queries that run over multiple tables/collections." Forgive my ignorance, but that is the whole point of working with a relational database. If cannot use JOINS then this solves only a very limited use case.

The biggest usecase for EventReduce is realtime applications. Most technologies for these like Firebase, AWS AppSync etc. work on non-relational data. If you want to use EventReduce with relational queries, you have to make them non-relational before, for example by using materialized views. If you do not want to do that, you should not use this algorithm in its current featureset.

thank you for the clarification

Re: EventReduce: An algorithm to optimize database queries that run multiple times

#56

Post by truth_seeker »

eventreduce wrote: post #42
truth_seeker wrote: post #35

IMO An open cursor of Change Stream with Aggregation pipeline (for given use-case) in MongoDB is more flexible solution to achieve this functionality. In addition, it also tracks the history of changes and hence allows the cursor to go back if needed with "resumeToken" https://docs.mongodb.com/manual/changeStreams/

There is a big difference between a change-stream and a realtime query. For example mongodbs cursor-stream is a good way to observe the events that happen to a specific collection or documents that match some criteria. If you want the realtime-results of a query that has sorting, skip limit etc. than it is really hard to warp the changestream into this. In fact this is exactly what EventReduce could do for you. For m…

>> There is a big difference between a change-stream and a realtime query. For example mongodbs cursor-stream is a good way to observe the events that happen to a specific collection or documents that match some criteria. If you want the realtime-results of a query that has sorting, skip limit etc. than it is really hard to warp the changestream into this.

Have you actually tried it from mongo shell or any mongodb client driver ?

In the official document link which i shared it is clearly mentioned it supports aggregation pipeline. Any operator which is compatible with aggregation pipeline framework including "$sort" and "$skip" can be used. You can also use JOIN like operator "$lookup" or "$graphLookup".

See this link for info https://docs.mongodb.com/manual/core/aggregation-pipeline-op...

Re: EventReduce: An algorithm to optimize database queries that run multiple times

#58

Post by ComodoHacker »

eventreduce wrote: post #41
ComodoHacker wrote: post #37

Earlier quoted context omitted.

And the main cost of this (questionable IMO) benefit is losing consistency, which is losing any change to DB not coming from the calling app. You haven't mentioned this cost anywhere.

The writes are not tunneled somehow through this algorithm. You still use the database like your normally would do. So the consistency is not affected. Also this is an open source project, not something I want to sell you. Feel free to make a PR/issue with any open topics that are not mentioned in the docs.

>The writes are not tunneled somehow through this algorithm

Then I fail to understand how it works. How Event-Reduce becomes aware of these "write events"?

>this is an open source project, not something I want to sell you

You made it open source so others can use it, right? They better be making an informed decision whether your solution suits their needs.

Re: EventReduce: An algorithm to optimize database queries that run multiple times

#59

Post by eventreduce »

truth_seeker wrote: post #56
eventreduce wrote: post #42

Earlier quoted context omitted.

There is a big difference between a change-stream and a realtime query. For example mongodbs cursor-stream is a good way to observe the events that happen to a specific collection or documents that match some criteria. If you want the realtime-results of a query that has sorting, skip limit etc. than it is really hard to warp the changestream into this. In fact this is exactly what EventReduce could do for you. For m…

>> There is a big difference between a change-stream and a realtime query. For example mongodbs cursor-stream is a good way to observe the events that happen to a specific collection or documents that match some criteria. If you want the realtime-results of a query that has sorting, skip limit etc. than it is really hard to warp the changestream into this. Have you actually tried it from mongo shell or any mongodb cl…

Yes I used it. I actually know it really well. I also did performance comparisons with mongodb and mongodbs change stream and cursors. What I posted here is just an algorithm. You could now compare it to mongodb (a product) and say it is a "more flexible solution" but I do not see the point in directly comparing it simply based on the documentation of both.

Re: EventReduce: An algorithm to optimize database queries that run multiple times

#60

Post by eventreduce »

ComodoHacker wrote: post #58
eventreduce wrote: post #41

Earlier quoted context omitted.

The writes are not tunneled somehow through this algorithm. You still use the database like your normally would do. So the consistency is not affected. Also this is an open source project, not something I want to sell you. Feel free to make a PR/issue with any open topics that are not mentioned in the docs.

>The writes are not tunneled somehow through this algorithm Then I fail to understand how it works. How Event-Reduce becomes aware of these "write events"? >this is an open source project, not something I want to sell you You made it open source so others can use it, right? They better be making an informed decision whether your solution suits their needs.

You have to provide the events by yourself. See EventReduce as a simple function that can do oldResults+Event=newResults.

And yes, you should always do testings before you use open source stuff. There is no warranty use it on your own risk.

Post reply on HN