Earlier quoted context omitted.
Worth mentioning Pekko, the Akka fork. https://pekko.apache.org/
Is this an active fork?
Seems like it based on GH activity, but I don’t know for sure.
Hydro: Distributed Programming Framework for Rust
21–30 of 50 posts
Re: Hydro: Distributed Programming Framework for Rust
#22Earlier quoted context omitted.
Worth mentioning Pekko, the Akka fork. https://pekko.apache.org/
Is this an active fork?
In 2022 Lightbend relicensed Akka from Apache 2.0 to BSL, which was a huge problem for all of the other opensource projects (like Flink) that used it as part of their coordination layer. At this point most or all of them have moved to Pekko, which is a fork of the last release of Akka under Apache 2.0.
Re: Hydro: Distributed Programming Framework for Rust
#23How does this compare to timely [0] in terms of data flow? Can you represent control flow like loops in the IR? [0] https://github.com/TimelyDataflow/timely-dataflow
Reading a bit about it from the Flo paper - Describe a dataflow graph just like Timely - Comes from a more "semantic dataflow" kind of heritage (frp, composition, flow-of-flows, algebraic operators, proof-oriented) as opposed to the more operationally minded background of Timely - Has a (very) different notion of "progress" than Timely, focused instead of ensuring the compositions are generative in light of potential…
Thanks for the summary, really looks like something that is worth digging into!
Re: Hydro: Distributed Programming Framework for Rust
#24Earlier quoted context omitted.
This is a project out of the riselab https://rise.cs.berkeley.edu/projects/ Most data processing and distributed systems have some sort of link back to the research this lab has done.
> Most data processing and distributed systems have some sort of link back to the research this lab has done. Heh. "most data processing and distributed systems"? Surely you don't mean that the rest of the world was sitting tight working on uniprocessors until this lab got set up in 2017!
I assume they're talking about the longer history of the distributed systems lab at Berkeley, which was AMP before RISE. (It's actually now Sky Lab[0], each of the labs live for 5 years). AMP notably is the origin of Spark, Mesos, and Tachyon (now Alluxio), and RISE originated Ray.
Re: Hydro: Distributed Programming Framework for Rust
#25If there is an intermediary language in the middle with its own runtime, does that mean that we lose everything Rust brings? I thought this would introduce the language to choreograph separate Rust binaries into a consistent and functional distributed system but it looks more like you're writing DFIR the whole way through and not just as glue
DFIR is implemented in Rust, if that's what you're asking.
Re: Hydro: Distributed Programming Framework for Rust
#26Earlier quoted context omitted.
> Most data processing and distributed systems have some sort of link back to the research this lab has done. Heh. "most data processing and distributed systems"? Surely you don't mean that the rest of the world was sitting tight working on uniprocessors until this lab got set up in 2017!
I assume they're talking about the longer history of the distributed systems lab at Berkeley, which was AMP before RISE. (It's actually now Sky Lab[0], each of the labs live for 5 years). AMP notably is the origin of Spark, Mesos, and Tachyon (now Alluxio), and RISE originated Ray. [0] https://sky.cs.berkeley.edu/
correct
Re: Hydro: Distributed Programming Framework for Rust
#27Love the effort but I would love an “akka.rs” to eventually make its way into the Rust ecosystem.
Be careful what you wish for!
Re: Hydro: Distributed Programming Framework for Rust
#28Earlier quoted context omitted.
> Most data processing and distributed systems have some sort of link back to the research this lab has done. Heh. "most data processing and distributed systems"? Surely you don't mean that the rest of the world was sitting tight working on uniprocessors until this lab got set up in 2017!
I assume they're talking about the longer history of the distributed systems lab at Berkeley, which was AMP before RISE. (It's actually now Sky Lab[0], each of the labs live for 5 years). AMP notably is the origin of Spark, Mesos, and Tachyon (now Alluxio), and RISE originated Ray. [0] https://sky.cs.berkeley.edu/
Right .. the AMPLab was set up in 2011. The Djikstra prize for distributed computing was set up in 2006 .. people like Djikstra and Lamport and Jim Gray and Barbara Liskov won Turing Awards for a lifetime's worth of work.
Now, Berkeley has been a fount of research on the topic, no question about that. I myself worked there (on Bloom, with Joe Hellerstein). But forgetting the other top universities of the world is a bit ... amusing?
Let's take one of the many lists of foundational papers of this field:
http://muratbuffalo.blogspot.com/2021/02/foundational-distri...
How many came out of Berkeley, let alone a recent entry like the AMPLab?
Re: Hydro: Distributed Programming Framework for Rust
#29Earlier quoted context omitted.
> Most data processing and distributed systems have some sort of link back to the research this lab has done. Heh. "most data processing and distributed systems"? Surely you don't mean that the rest of the world was sitting tight working on uniprocessors until this lab got set up in 2017!
I assume they're talking about the longer history of the distributed systems lab at Berkeley, which was AMP before RISE. (It's actually now Sky Lab[0], each of the labs live for 5 years). AMP notably is the origin of Spark, Mesos, and Tachyon (now Alluxio), and RISE originated Ray. [0] https://sky.cs.berkeley.edu/
There is a nice article by David Patterson (who used to direct the lab and won the Turing Award) on why Berkeley changes the name and scope of the lab every five years https://www2.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-... . Unfortunately, there's no good name for the lab across each of the five-year boundaries so people just say "rise lab" or "amp lab" etc.
Re: Hydro: Distributed Programming Framework for Rust
#30If there is an intermediary language in the middle with its own runtime, does that mean that we lose everything Rust brings? I thought this would introduce the language to choreograph separate Rust binaries into a consistent and functional distributed system but it looks more like you're writing DFIR the whole way through and not just as glue
Hi, I'm one of the PhD students leading the work on Hydro!
DFIR is more of a middle-layer DSL that allows us (the high-level language developers) to re-structure your Rust code to make it more amenable to low-level optimizations like vectorization. Because DFIR operators (like map, filter, etc.) take in Rust closures, we can pass those through all the way from the high-level language to the final Rust binaries. So as a user, you never interact with DFIR.