Differential Dataflow for the Masses

#1

Post by rebanevapustus » Fri, Nov 08, 2024, 7:45 PM UTC

Differential Dataflow for the Masses
github.com

Re: Differential Dataflow for the Masses

#2

Post by snthpy » Sat, Nov 09, 2024, 8:13 AM UTC

Love this project!

Since "Differential Dataflow" has a specific meaning though and this isn't it, I wonder if a different title wouldn't be better?

"DBSP for the masses" presumes someone already knows what DBSP is so that doesn't work.

What about "Incremental Dataflow for the masses" or "Incremental Dataflow in Python"?

Re: Differential Dataflow for the Masses

#3

Post by snthpy » Sat, Nov 09, 2024, 8:18 AM UTC

Also, I never fully grokked Differential Dataflow but I get DBSP so I would love an explanation of how Differential Dataflow differs from DBSP.

Re: Differential Dataflow for the Masses

#4

Post by rebanevapustus » Sat, Nov 09, 2024, 9:01 AM UTC

snthpy wrote: post #3

Also, I never fully grokked Differential Dataflow but I get DBSP so I would love an explanation of how Differential Dataflow differs from DBSP.

https://github.com/brurucy/pydbsp/issues/6 it's been answered! (to the best of my knowledge)

Re: Differential Dataflow for the Masses

#5

Post by rebanevapustus » Sat, Nov 09, 2024, 9:01 AM UTC

snthpy wrote: post #2

Love this project! Since "Differential Dataflow" has a specific meaning though and this isn't it, I wonder if a different title wouldn't be better? "DBSP for the masses" presumes someone already knows what DBSP is so that doesn't work. What about "Incremental Dataflow for the masses" or "Incremental Dataflow in Python"?

Incremental Dataflow for the Masses sounds much better :)

Re: Differential Dataflow for the Masses

#6

Post by keithalewis » Wed, Nov 13, 2024, 9:15 PM UTC

This seems to make the mistake of identifying a vector with a stream.

Re: Differential Dataflow for the Masses

#7

Post by rebanevapustus » Wed, Nov 13, 2024, 11:30 PM UTC

keithalewis wrote: post #6

This seems to make the mistake of identifying a vector with a stream.

In this implementation streams are not vectors nor lists. They are ordered hash maps.

Re: Differential Dataflow for the Masses

#8

Post by erichocean » Thu, Nov 14, 2024, 5:14 PM UTC

Electric Clojure v3 also implements differential dataflow. [0]

[0] https://www.youtube.com/watch?v=mP9e_FDjHLU

Re: Differential Dataflow for the Masses

#9

Post by rebanevapustus » Thu, Nov 14, 2024, 10:04 PM UTC

erichocean wrote: post #8

Electric Clojure v3 also implements differential dataflow. [0] [0] https://www.youtube.com/watch?v=mP9e_FDjHLU

Electric Clojure v3 cannot be "Differential Dataflow" because it *seems* to be restricted to DAGs only. Differential Dataflow is highly expressive, it supports iteration.

Re: Differential Dataflow for the Masses

#10

Post by dustingetz » Thu, Nov 14, 2024, 10:42 PM UTC

rebanevapustus wrote: post #9

erichocean wrote: post #8

Electric Clojure v3 also implements differential dataflow. [0] [0] https://www.youtube.com/watch?v=mP9e_FDjHLU

Electric Clojure v3 cannot be "Differential Dataflow" because it *seems* to be restricted to DAGs only. Differential Dataflow is highly expressive, it supports iteration.

(electric founder here) we are lowercase differential dataflow my friend. Fwiw our model does support cycles and iteration, we simply didn’t bother to formally reify cycles yet, because we have a functional effect system under the hood and using that to implement cycles by userland side effect does the right thing, is perfectly efficient, and has very little downside.