Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feldera Incremental Compute Engine

github.com

Post reply on HN
Print view View original

21–30 of 56 posts

Re: Feldera Incremental Compute Engine

#21

Post by lsuresh »

bbminner wrote: post #18

I wonder what guarantees can be made wrt resource consumption. I suppose that'd reasonable to assume that in most (all?) cases an update is cheaper then recompute in terms of cpu cycles, but what about ram? Intuitively it seems like there must be cases that would force you to store unbounded amount of data indefinitely in ram.

Your intuition is correct. Incremental computation is fundamentally a time-space tradeoff. Depending on the views you write, you might end up maintaining large amounts of state. We've written about it here: https://www.feldera.com/blog/streaming-needs-storage

That said, Feldera has several features to keep state bounded even when computing on infinite streams. For example, we do automatic garbage collection (GC) where with some static analysis, we can figure out when it is safe to forget inputs that will no longer affect the output of views.

We recently ported a community member's warehouse workload to Feldera where with these features, we were evaluating As-Of joins and streaming aggregations with 1.2GB of RAM on a laptop with more than a million events/sec in perf.

Re: Feldera Incremental Compute Engine

#22

Post by lsuresh »

qazxcvbnm wrote: post #19

Incredible… I hadn’t even noticed, and people found the holy grail and open-sourced it! By the way, I was wondering about a related question. Do streaming engines typically store a copy of the data streamed to them? For instance, if I had a view to get the maximum value of a table, and the maximum value was removed, the streaming engine surely needs to get the next value from somewhere. It seems clear that the stream…

Thank you!

The state Feldera maintains depends on the queries you write and the working set or windows you're computing over. Any time there are joins, distinct or non-linear aggregations, we need to maintain state as you've guessed.

A cool feature in Feldera is that it can compute over infinite streams with finite state because we automate garbage collection. The user specifies lateness over data sources or even views, and with some static analysis, Feldera determines when it is safe to forget old state such that it won't affect the output of any views.

Re: Feldera Incremental Compute Engine

#23

Post by lsuresh »

cube2222 wrote: post #15
lsuresh wrote: post #10

Earlier quoted context omitted.

Thanks for the kind words! (Feldera's CEO here) - We evaluate top-k queries incrementally and the nesting shouldn't be a problem for the engine (or it'd be a bug). If you have an example of a query, we can try it out at our end. - Yes. It is internally consistent. We've verified with the experiment here: https://www.scattered-thoughts.net/writing/internal-consiste... . Our guarantee is that we always produce the same…

Thanks for the response and clarifications! I think this scenario would illustrate it. Make a table with one column, x, and insert into it rows with values 1-5, and then 8-20. Then query it using more or less `SELECT x FROM (SELECT x FROM xs LIMIT 15 ORDER BY x) LIMIT 10`, and then insert 6 into the table. Output should be 1-6, 8-11. Of course as long as the limits aren't merged together during optimisation, that wou…

Thanks! Looks like that works.

Here is the query I set up on try.feldera.com.

  CREATE TABLE foo (x INTEGER NOT NULL PRIMARY KEY) WITH ('materialized' = 'true') ;

  CREATE MATERIALIZED VIEW bar AS SELECT x FROM (SELECT x FROM foo ORDER BY x LIMIT 15) LIMIT 10;
I then used our CLI tool fda to insert some rows and inspect the states after starting the pipeline: https://docs.feldera.com/reference/cli

  try.feldera.com/foo> select * from foo;

  +----+
  | x  |
  +----+
  | 1  |
  | 2  |
  | 3  |
  | 4  |
  | 5  |
  | 8  |
  | 9  |
  | 10 |
  | 11 |
  | 12 |
  | 13 |
  | 14 |
  | 15 |
  | 16 |
  | 17 |
  | 18 |
  | 19 |
  | 20 |
  +----+

  try.feldera.com/foo> insert into foo values (6);

  +-------+
  | count |
  +-------+
  | 1     |
  +-------+

  try.feldera.com/foo> select * from bar;

  +----+
  | x  |
  +----+
  | 1  |
  | 2  |
  | 3  |
  | 4  |
  | 5  |
  | 6  |
  | 8  |
  | 9  |
  | 10 |
  | 11 |
  +----+

Re: Feldera Incremental Compute Engine

#24

Post by lsuresh »

jitl wrote: post #17

I’ve been following the Feldera/DBSP/Differential Datalog team for a while and am happy to see y’all stable-ish with your own venture and settling in a model more approachable than DDlog for most developers :) This seems much more adoptable to me in my org than DDlog was, even if I really liked DDlog much more than SQL :-(

Thanks for following our journey! There's still room for more language frontends if you'd like to contribute. :)

Re: Feldera Incremental Compute Engine

#25

Post by lsuresh »

loxias wrote: post #16
Nelkins wrote: post #4

I would love if something like this that exposed C bindings so that every language with an FFI could use the library. I’d love to be able to define pipelines and queries in .NET instead of having to use SQL.

Second the desire for C bindings! (or someone showing how to wrap and call the rust bindings?)

The previous implementation we built at VMware went from datalog -> Rust, and we supported other language bindings using C bindings and FFI. The same ought to work here too.

Re: Feldera Incremental Compute Engine

#26

Post by cube2222 »

lsuresh wrote: post #23
cube2222 wrote: post #15

Earlier quoted context omitted.

Thanks for the response and clarifications! I think this scenario would illustrate it. Make a table with one column, x, and insert into it rows with values 1-5, and then 8-20. Then query it using more or less `SELECT x FROM (SELECT x FROM xs LIMIT 15 ORDER BY x) LIMIT 10`, and then insert 6 into the table. Output should be 1-6, 8-11. Of course as long as the limits aren't merged together during optimisation, that wou…

Thanks! Looks like that works. Here is the query I set up on try.feldera.com. CREATE TABLE foo (x INTEGER NOT NULL PRIMARY KEY) WITH ('materialized' = 'true') ; CREATE MATERIALIZED VIEW bar AS SELECT x FROM (SELECT x FROM foo ORDER BY x LIMIT 15) LIMIT 10; I then used our CLI tool fda to insert some rows and inspect the states after starting the pipeline: https://docs.feldera.com/reference/cli try.feldera.com/foo> se…

Awesome, thanks for double-checking!

Re: Feldera Incremental Compute Engine

#27

Post by jacques_chester »

I remember seeing a VMware-internal presentation on the DDlog work which led to Feldera and being absolutely blown away. They took a stream processing problem that had grown to an hours-deep backlog and reduced it to sub second processing times. Lalith & co are the real deal.

Re: Feldera Incremental Compute Engine

#28

Post by lsuresh »

jacques_chester wrote: post #27

I remember seeing a VMware-internal presentation on the DDlog work which led to Feldera and being absolutely blown away. They took a stream processing problem that had grown to an hours-deep backlog and reduced it to sub second processing times. Lalith & co are the real deal.

Thank you jacques_chester! Piping all that credit to my co-founders Mihai and Leonid, the key inventors.

Re: Feldera Incremental Compute Engine

#30

Post by rebanevapustus »

Big fan of Feldera here.

I would advise everybody to stay clear of anything that isn't Feldera or Materialize. Nobody aside from these guys have a IVM product that is grounded on proper theory.

If you are interested in trying out the theory (DBSP) underneath Feldera, but in Python, then check this out: https://github.com/brurucy/pydbsp

It works with pandas, polars...anything.

Post reply on HN