Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hydro: Distributed Programming Framework for Rust

hydro.run

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Hydro: Distributed Programming Framework for Rust

#41

Post by sitkack »

sriram_malhar wrote: post #28
necubi wrote: post #24

Earlier quoted context omitted.

I assume they're talking about the longer history of the distributed systems lab at Berkeley, which was AMP before RISE. (It's actually now Sky Lab[0], each of the labs live for 5 years). AMP notably is the origin of Spark, Mesos, and Tachyon (now Alluxio), and RISE originated Ray. [0] https://sky.cs.berkeley.edu/

Right .. the AMPLab was set up in 2011. The Djikstra prize for distributed computing was set up in 2006 .. people like Djikstra and Lamport and Jim Gray and Barbara Liskov won Turing Awards for a lifetime's worth of work. Now, Berkeley has been a fount of research on the topic, no question about that. I myself worked there (on Bloom, with Joe Hellerstein). But forgetting the other top universities of the world is a b…

You are mischaracterizing my comment, what I said was true. Most distributed systems work (now) has a link back to Berkeley distributed systems labs. Someone wanted context about Hydro (Joe Hellerstein).

I am not going to make every contextualizing comment an authoritative bibliography , you of all people could have added that w/o being snarky and starting this whole subthread.

Re: Hydro: Distributed Programming Framework for Rust

#42

Post by IshKebab »

shadaj wrote: post #31
IshKebab wrote: post #13

This could do with some real-world application examples so I can understand where you might want to apply it.

These code examples aren't fully documented yet (which is why we've not linked them in the documentation), but you can take a look at a (more-real) implementation of Paxos here: https://github.com/hydro-project/hydro/blob/main/hydro_test/... . We're also working on building more complex applications like a key-value store.

I meant concrete real-world applications like "a WhatsApp clone" or whatever. Paxos is very abstract. Nobody deploys "a paxos".

Re: Hydro: Distributed Programming Framework for Rust

#44

Post by weego »

thelittlenag wrote: post #27
halfmatthalfcat wrote: post #15

Love the effort but I would love an “akka.rs” to eventually make its way into the Rust ecosystem.

Be careful what you wish for!

We appear to be wishing for writing tons of boilerplate with nonsensical use of implicits

Re: Hydro: Distributed Programming Framework for Rust

#45

Post by rozap »

weego wrote: post #44
thelittlenag wrote: post #27

Earlier quoted context omitted.

Be careful what you wish for!

We appear to be wishing for writing tons of boilerplate with nonsensical use of implicits

i once heard it described as "erlang except it's bad and 30 years later on the JVM" which, ime, is harsh but accurate

Re: Hydro: Distributed Programming Framework for Rust

#46

Post by FridgeSeal »

tomnicholas1 wrote: post #9
benrutter wrote: post #4

Earlier quoted context omitted.

From first glance it looks conceptually pretty similar to some work in the data-science space, I'm thinking of spark (which they mention in their docs) and dask. My knee-jerk excitements is that this has the potential to be pretty powerful specifically because it's based on Rust so can play really nicely with other languages. Spark runs on the JVM which is a good choice for portability but still introduces a bunch of…

I was also going to say this looks similar to one layer of dask - dask takes arbitrary python code and uses cloudpickle to serialise it in order to propagate dependencies to workers, this seems to be an equivalent layer for rust.

This looks to be a degree more sophisticated than that.

Authors in the comments here mention that the flo compiler (?) will accept-and-rewrite Rust code to make it more amenable to distribution. It also appears to be building and optimising the data-flow rather than just distributing the work. There’s also comparisons to timely, which I believe does some kind of incremental compute.

Re: Hydro: Distributed Programming Framework for Rust

#47

Post by irq-1 »

conor-23 wrote: post #29
necubi wrote: post #24

Earlier quoted context omitted.

I assume they're talking about the longer history of the distributed systems lab at Berkeley, which was AMP before RISE. (It's actually now Sky Lab[0], each of the labs live for 5 years). AMP notably is the origin of Spark, Mesos, and Tachyon (now Alluxio), and RISE originated Ray. [0] https://sky.cs.berkeley.edu/

There is a nice article by David Patterson (who used to direct the lab and won the Turing Award) on why Berkeley changes the name and scope of the lab every five years https://www2.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-... . Unfortunately, there's no good name for the lab across each of the five-year boundaries so people just say "rise lab" or "amp lab" etc.

Interesting.

> Good Commandment 3. Thou shalt limit the duration of a center. ...

> To hit home runs, it’s wise to have many at bats. ...

> It’s hard to predict information technology trends much longer than five years. ...

> US Graduate student lifetimes are about five years. ...

> You need a decade after a center finishes to judge if it was a home run. Just 8 of the 12 centers in Table I are old enough, and only 3 of them—RISC, RAID, and the Network of Workstations center—could be considered home runs. If slugging .375 is good, then I’m glad that I had many 5-­‐year centers rather than fewer long ones.

(Network of Workstations > Google)

Re: Hydro: Distributed Programming Framework for Rust

#48

Post by sriram_malhar »

sitkack wrote: post #41
sriram_malhar wrote: post #28

Earlier quoted context omitted.

Right .. the AMPLab was set up in 2011. The Djikstra prize for distributed computing was set up in 2006 .. people like Djikstra and Lamport and Jim Gray and Barbara Liskov won Turing Awards for a lifetime's worth of work. Now, Berkeley has been a fount of research on the topic, no question about that. I myself worked there (on Bloom, with Joe Hellerstein). But forgetting the other top universities of the world is a b…

You are mischaracterizing my comment, what I said was true. Most distributed systems work (now) has a link back to Berkeley distributed systems labs. Someone wanted context about Hydro (Joe Hellerstein). I am not going to make every contextualizing comment an authoritative bibliography , you of all people could have added that w/o being snarky and starting this whole subthread.

> Most distributed systems work (now) has a link back to Berkeley distributed systems labs.

I didn't think you were saying that most distributed systems work happening at Berkeley harks back to earlier work at Berkeley. That's a bit obvious.

The only way I can interpret "most distributed systems work now" is a statement about work happening globally. In which case it is a sweeping and false generalization.

Is there another interpretation?

Re: Hydro: Distributed Programming Framework for Rust

#49

Post by pradn »

Paradigma11 wrote: post #7
djtango wrote: post #3

This is really exciting. Is anyone familiar with this space able to point to prior art? Have people built similar frameworks in other languages? I know different people have worked on dataflow and remember thinking Materialize was very cool and I've used Kafka Streams at work before, and I remember thinking that a framework probably made sense for stitching this all together

It looks like a mixture between Akka ( https://getakka.net/ less enterprisy than the Java version), which is based on the actor model and has a focus on distributed systems, and reactive libraries like rx ( https://reactivex.io/ ). So maybe https://doc.akka.io/libraries/akka-core/current/stream/index... is the best fit.

An important design consideration for Hydro, it seems, is to be able to define a workflow in a higher level language and then be able to cut them into different binaries.

Is that something Akka / RX offer? My quick thought is that they structure code in one binary.

Re: Hydro: Distributed Programming Framework for Rust

#50

Post by sebstefan »

cess11 wrote: post #25
sebstefan wrote: post #17

If there is an intermediary language in the middle with its own runtime, does that mean that we lose everything Rust brings? I thought this would introduce the language to choreograph separate Rust binaries into a consistent and functional distributed system but it looks more like you're writing DFIR the whole way through and not just as glue

DFIR is implemented in Rust, if that's what you're asking.

I can implement Lua in Rust and lose everything Rust brings when I code in Lua.
Post reply on HN