Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hydro: Distributed Programming Framework for Rust

hydro.run

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Hydro: Distributed Programming Framework for Rust

#11

Post by mplanchard »

the_duke wrote: post #10

So each "process" is deployed as a separate binary, so presumably run as a separate process? If so, this seems somewhat problematic in terms of increased overhead. How is fast communication achieved? Some fast shared memory IPC mechanism? Also, I don't see anything about integration with async? For better or worse, the overwhelming majority of code dealing with networking has migrated to async. You won't find good no…

By “distributed” I assumed it meant “distributed,”as in on entirely separate machines. Thus necessitating each component running as an independent process.

Re: Hydro: Distributed Programming Framework for Rust

#12

Post by leicmi »

stefanka wrote: post #2

How does this compare to timely [0] in terms of data flow? Can you represent control flow like loops in the IR? [0] https://github.com/TimelyDataflow/timely-dataflow

Their latest paper [0] refers to Naiad(timely dataflow) a few times, e.g.: "Inspired by ingress/egress nodes in Naiad [34], nested streams can be processed by nested dataflow graphs, which iteratively process chunks of data sourced from a larger stream with support for carrying state across iterations."

[0] https://hydro.run/papers/flo.pdf

Re: Hydro: Distributed Programming Framework for Rust

#14

Post by Cyph0n »

Paradigma11 wrote: post #7
djtango wrote: post #3

This is really exciting. Is anyone familiar with this space able to point to prior art? Have people built similar frameworks in other languages? I know different people have worked on dataflow and remember thinking Materialize was very cool and I've used Kafka Streams at work before, and I remember thinking that a framework probably made sense for stitching this all together

It looks like a mixture between Akka ( https://getakka.net/ less enterprisy than the Java version), which is based on the actor model and has a focus on distributed systems, and reactive libraries like rx ( https://reactivex.io/ ). So maybe https://doc.akka.io/libraries/akka-core/current/stream/index... is the best fit.

Worth mentioning Pekko, the Akka fork.

https://pekko.apache.org/

Re: Hydro: Distributed Programming Framework for Rust

#16

Post by sriram_malhar »

sitkack wrote: post #6
djtango wrote: post #3

This is really exciting. Is anyone familiar with this space able to point to prior art? Have people built similar frameworks in other languages? I know different people have worked on dataflow and remember thinking Materialize was very cool and I've used Kafka Streams at work before, and I remember thinking that a framework probably made sense for stitching this all together

This is a project out of the riselab https://rise.cs.berkeley.edu/projects/ Most data processing and distributed systems have some sort of link back to the research this lab has done.

> Most data processing and distributed systems have some sort of link back to the research this lab has done.

Heh. "most data processing and distributed systems"? Surely you don't mean that the rest of the world was sitting tight working on uniprocessors until this lab got set up in 2017!

Re: Hydro: Distributed Programming Framework for Rust

#17

Post by sebstefan »

If there is an intermediary language in the middle with its own runtime, does that mean that we lose everything Rust brings?

I thought this would introduce the language to choreograph separate Rust binaries into a consistent and functional distributed system but it looks more like you're writing DFIR the whole way through and not just as glue

Re: Hydro: Distributed Programming Framework for Rust

#18

Post by haolez »

Cyph0n wrote: post #14
Paradigma11 wrote: post #7

Earlier quoted context omitted.

It looks like a mixture between Akka ( https://getakka.net/ less enterprisy than the Java version), which is based on the actor model and has a focus on distributed systems, and reactive libraries like rx ( https://reactivex.io/ ). So maybe https://doc.akka.io/libraries/akka-core/current/stream/index... is the best fit.

Worth mentioning Pekko, the Akka fork. https://pekko.apache.org/

Is this an active fork?

Re: Hydro: Distributed Programming Framework for Rust

#19

Post by tel »

stefanka wrote: post #2

How does this compare to timely [0] in terms of data flow? Can you represent control flow like loops in the IR? [0] https://github.com/TimelyDataflow/timely-dataflow

Reading a bit about it from the Flo paper

- Describe a dataflow graph just like Timely - Comes from a more "semantic dataflow" kind of heritage (frp, composition, flow-of-flows, algebraic operators, proof-oriented) as opposed to the more operationally minded background of Timely - Has a (very) different notion of "progress" than Timely, focused instead of ensuring the compositions are generative in light of potentially unbounded streaming inputs - In fact, Flo doesn't really have any notion of "timeliness", no timestamping at all - Supports nested looping like Timely, though via a very different mechanism. The basic algebra is extremely non-cyclic, but the nested streams/graphs formalism allows for iteration.

The paper also makes a direct comparison with DBSP, which as I understand it, is also part of the Timely/Naiad heritage. Similar to Timely, the authors suggest that Flo could be a unifying semantic framework for several other similar systems (Flink, LVars, DBSP).

So I'd say that the authors of Flo are aware of Naiad/Timely and took inspiration of nested iterative graphs, but little else.

Re: Hydro: Distributed Programming Framework for Rust

#20

Post by macintux »

sriram_malhar wrote: post #16
sitkack wrote: post #6

Earlier quoted context omitted.

This is a project out of the riselab https://rise.cs.berkeley.edu/projects/ Most data processing and distributed systems have some sort of link back to the research this lab has done.

> Most data processing and distributed systems have some sort of link back to the research this lab has done. Heh. "most data processing and distributed systems"? Surely you don't mean that the rest of the world was sitting tight working on uniprocessors until this lab got set up in 2017!

I too was a bit surprised by the assertion, but it doesn't say "ancestry", just "link".

And I'm guessing if you include BOOM[1], the links are even deeper.

[1] http://boom.cs.berkeley.edu

Post reply on HN