31–40 of 50 posts
This could do with some real-world application examples so I can understand where you might want to apply it.
This is really exciting. Is anyone familiar with this space able to point to prior art? Have people built similar frameworks in other languages? I know different people have worked on dataflow and remember thinking Materialize was very cool and I've used Kafka Streams at work before, and I remember thinking that a framework probably made sense for stitching this all together
From first glance it looks conceptually pretty similar to some work in the data-science space, I'm thinking of spark (which they mention in their docs) and dask. My knee-jerk excitements is that this has the potential to be pretty powerful specifically because it's based on Rust so can play really nicely with other languages. Spark runs on the JVM which is a good choice for portability but still introduces a bunch of…
So each "process" is deployed as a separate binary, so presumably run as a separate process? If so, this seems somewhat problematic in terms of increased overhead. How is fast communication achieved? Some fast shared memory IPC mechanism? Also, I don't see anything about integration with async? For better or worse, the overwhelming majority of code dealing with networking has migrated to async. You won't find good no…
At POPL 2025 (last week!), an undergraduate working on Hydro presented a compiler that automatically compiles blocks of async-await code into Hydro dataflow. You can check out that (WIP, undocumented) compiler here: https://github.com/hydro-project/HydraulicLift
How does this compare to timely [0] in terms of data flow? Can you represent control flow like loops in the IR? [0] https://github.com/TimelyDataflow/timely-dataflow
Reading a bit about it from the Flo paper - Describe a dataflow graph just like Timely - Comes from a more "semantic dataflow" kind of heritage (frp, composition, flow-of-flows, algebraic operators, proof-oriented) as opposed to the more operationally minded background of Timely - Has a (very) different notion of "progress" than Timely, focused instead of ensuring the compositions are generative in light of potential…
This is also one of the core differences of Timely compared to DBSP, which uses a flat representation (z-sets) to store retractions rather than using versioned elements. This allows retractions to be propagated as just negative item counts which fits into the Flo model (and therefore Hydro).
looks really cool and I can see a few ways how to use it, especially deploy part which seems unique. Looking forward to more fleshed-out documentation, especially seemingly crucial Streams and Singletons and Optionals part.
https://www.youtube.com/watch?v=YpMKUQKlak0&ab_channel=ACMSI...
How does this compare to using something like Ballista for data pipelines? The latter benefits a lot from building on top of Apache Arrow and Apache Datafusion.