Viewing profile — frankmcsherry

> Sounds like basic memoization and topological sort gets you all the way there? I don't really want to pull rank here, but for the benefit of other readers: 100% nope. I personall…

I think maybe they were confused by this text (which I agree has nothing to do with Rust itself breaking): > With the release of Apache Arrow 3.0.0 there are many breaking changes …

> If SQL had a way of picking one row from a group, rather than aggregating over it, that would be immensely useful. You can do this with a LATERAL join, if you want to avoid the j…

TD lets you write whatever logic you want (it is fairly unopinionated on your logic and state). Differential dataflow plugs in certain logic there, and it does indeed maintain a sy…

It just comes down to something as simple as: "if I have shown you 1M different things, and now show you one more thing, what do you have to do to tell me whether that one thing is…

There are a few differences, the main one between Spark and timely dataflow is that TD operators can be stateful, and so can respond to new rounds of input data in time proportiona…

I think the main distinction is around "interactivity" and how long it takes from typing a query to getting results out. Once you stand up a Flink dataflow, it should move along a …

It's a good question, but you'd have to ask them I think. Tamas (from Itemis) and I were in touch for a while, mostly shaking out why DD was out-performing their previous approach,…

Here's my take on this, from a few months back: https://materialize.com/lateral-joins-and-demand-driven-quer...

Min and max work using a hierarchical reduction tree, the dataflow equivalent of a priority queue. They will update, under arbitrary changes to the input relation, in time proporti…

All queries are incrementally maintained with the property that we do work proportional to the number of records in difference at each intermediate stage of the query plan. That in…

I don't think there is anything fundamentally different from an existing database. In all relational databases, some lateral joins can be expensive to compute. In Materialize, thos…

> I take this to mean that Materialize cannot efficiently maintain a view with lateral joins [...] Well, no this isn't a correct take. Lateral joins introduce what is essentially a…

It's easier to describe the things that cannot be materialized. The only rule at the moment is that you cannot currently maintain queries that use the functions `current_time()`, `…

Hi! I work at Materialize. I think the right starter take is that Materialize is a deterministic compute engine, one that relies on other infrastructure to act as the source of tru…

Hi, I work at Materialize. You can read about Vertica's "Live Aggregate Projections" here: https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/An... In particular, there are …

> Data anonymized with Amnesia are statistically guaranteed that they cannot be linked to the original data. It looks like (from other text on their site) they use variants on k-an…

Naming things is hard. In this case, the command does not create a new dataflow, it creates a new timely dataflow computation by executing the closure on multiple workers. The comp…

It's a bit different, right, because that paper presents better algorithms on the same datasets as the work it cites.

Yeah, for example I have an open PR in differential dataflow to make some changes to avoid a planned break in the future. https://github.com/TimelyDataflow/differential-dataflow/pu…

> That's not accurate. Fwiw, I recorded at least four different times a Rust release broke timely / differential dataflow, and have seen a few others in other folks' code. Afaict, …

In SQL you would most likely be directed to use `WITH RECURSIVE`, which is something we plan to do, but not yet. It can be a bit gross to use WITH RECURSIVE, because there are ofte…

By moving up the stack a bit (managing computation, rather than storage) we can provide consistency using techniques other than just using the guarantees provided by the storage it…

> That title goes to "no tracking" This is roughly analogous to "abstinence is the best form of birth control". It's not wrong, but it also isn't particularly realistic or helpful.…