Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feldera Incremental Compute Engine

github.com

Post reply on HN
Print view View original

11–20 of 56 posts

Re: Feldera Incremental Compute Engine

#11

Post by lsuresh »

jonstewart wrote: post #7

How does it compare to Materialize/differential dataflow?

(Feldera's CEO here)

We are based on DBSP (https://www.vldb.org/pvldb/vol16/p1601-budiu.pdf) which is an evolution of DD. DBSP gives us an algorithm to take arbitrarily complex queries and generate an incremental version of it.

As a consequence, we evaluate everything incrementally. For example, we are the only engine that can perform rolling aggregates incrementally. In general, with DBSP, we can incrementalize "the full relational algebra, queries over sets and multisets, arbitrarily nested relations, aggregation, flatmap (unnest), monotonic and non-monotonic recursion, streaming aggregation, and arbitrary compositions of all of these". DBSP is a much simpler and cleaner foundation.

As a product, both our enterprise and open-source (MIT licensed) offerings let you run it anywhere you want including your laptop.

Positioning wise, we are a query engine with a unified way to compute over both bounded and unbounded data sources with perfect consistency, with an integrated storage layer. Materialize is going for building an operational warehouse.

Re: Feldera Incremental Compute Engine

#13

Post by arn3n »

If you don’t want to change your whole stack, ClickHouse’s Materialized Views do something extraordinarily similar, where computations are ran on inserts to the source table in an online/streaming manner. I’m curious how this solution compares in its set of features/gaurantees.

Re: Feldera Incremental Compute Engine

#14

Post by lsuresh »

arn3n wrote: post #13

If you don’t want to change your whole stack, ClickHouse’s Materialized Views do something extraordinarily similar, where computations are ran on inserts to the source table in an online/streaming manner. I’m curious how this solution compares in its set of features/gaurantees.

For incremental computation, Feldera is just way more powerful and general. It can evaluate arbitrarily sophisticated SQL programs incrementally (tables and deeply nested layers of views). For example, it can do rolling aggregates over joins, handle late and out-of-order arrivals, can compute over infinite streams with finite state (via automatic garbage collection), and it's strongly consistent. Clickhouse's materialized views are much simpler and restricted in comparison.

That said, we are /not/ a replacement ever for Clickhouse or any other historical warehouse. In fact, we pair best with one of them. Have data flow through or teed into Feldera to maintain sophisticated standing queries -- maintain historical data in your warhehouse.

Re: Feldera Incremental Compute Engine

#15

Post by cube2222 »

lsuresh wrote: post #10
cube2222 wrote: post #3

This looks extremely cool. This is basically incremental view maintenance in databases, a problem that almost everybody (I think) has when using SQL databases and wanting to do some derived views for more performant access patterns. Importantly, they seem to support a wide breath of SQL operators, support spilling computation state to disk, and it's open-source! Interestingly, it compiles queries to Rust, so an appro…

Thanks for the kind words! (Feldera's CEO here) - We evaluate top-k queries incrementally and the nesting shouldn't be a problem for the engine (or it'd be a bug). If you have an example of a query, we can try it out at our end. - Yes. It is internally consistent. We've verified with the experiment here: https://www.scattered-thoughts.net/writing/internal-consiste... . Our guarantee is that we always produce the same…

Thanks for the response and clarifications!

I think this scenario would illustrate it.

Make a table with one column, x, and insert into it rows with values 1-5, and then 8-20.

Then query it using more or less `SELECT x FROM (SELECT x FROM xs LIMIT 15 ORDER BY x) LIMIT 10`, and then insert 6 into the table. Output should be 1-6, 8-11. Of course as long as the limits aren't merged together during optimisation, that would make the test-case moot.

Good luck with your product!

Re: Feldera Incremental Compute Engine

#16

Post by loxias »

Nelkins wrote: post #4

I would love if something like this that exposed C bindings so that every language with an FFI could use the library. I’d love to be able to define pipelines and queries in .NET instead of having to use SQL.

Second the desire for C bindings! (or someone showing how to wrap and call the rust bindings?)

Re: Feldera Incremental Compute Engine

#17

Post by jitl »

I’ve been following the Feldera/DBSP/Differential Datalog team for a while and am happy to see y’all stable-ish with your own venture and settling in a model more approachable than DDlog for most developers :)

This seems much more adoptable to me in my org than DDlog was, even if I really liked DDlog much more than SQL :-(

Re: Feldera Incremental Compute Engine

#18

Post by bbminner »

I wonder what guarantees can be made wrt resource consumption. I suppose that'd reasonable to assume that in most (all?) cases an update is cheaper then recompute in terms of cpu cycles, but what about ram? Intuitively it seems like there must be cases that would force you to store unbounded amount of data indefinitely in ram.

Re: Feldera Incremental Compute Engine

#19

Post by qazxcvbnm »

Incredible… I hadn’t even noticed, and people found the holy grail and open-sourced it!

By the way, I was wondering about a related question. Do streaming engines typically store a copy of the data streamed to them? For instance, if I had a view to get the maximum value of a table, and the maximum value was removed, the streaming engine surely needs to get the next value from somewhere. It seems clear that the streaming engine needs at least its own snapshot of the data to have a consistent state of the computation, but duplicating the persisted data seems somewhat wasteful.

Re: Feldera Incremental Compute Engine

#20

Post by ben_pfaff »

bbminner wrote: post #18

I wonder what guarantees can be made wrt resource consumption. I suppose that'd reasonable to assume that in most (all?) cases an update is cheaper then recompute in terms of cpu cycles, but what about ram? Intuitively it seems like there must be cases that would force you to store unbounded amount of data indefinitely in ram.

(Feldera co-founder here.) There are some cases where Feldera needs to index data indefinitely, yes. For those cases, Feldera can put those indexes on storage rather than keeping them entirely in RAM.

In a lot of cases where one might initially think that data needs to stay around indefinitely, people actually want the results from the last hour or day or month, etc. For those cases, Feldera supports a concept called "lateness" that allows it to drop older data: https://docs.feldera.com/sql/streaming/#lateness-expressions.

Post reply on HN