I assume this is similar to Ray?
There's also Dask, which can do distributed pandas and numpy operations etc. However it was originally developed for traditional HPC systems and has only limited support for GPU computing. https://www.dask.org/
PyTorch Monarch
21–30 of 45 posts
Re: PyTorch Monarch
#22I made my own single controller PyTorch extension [1], though mines doesn't yet support cross node communication. I found it interesting to compare how Monarch makes things performant. I believe Monarch also uses cloudpickle for code to be shared among all nodes, which is probably the only way to performantly have various nodes execute work as that ends up being a one time setup cost. I found the fanning out of sendi…
> As far as things that might be a performance loss here, one thing I'm wondering is if custom kernels are supported
Yeah, you might end up needing some changes to remote worker initialization, but you can generally bake in whatever kernels and other system code you need.
Re: PyTorch Monarch
#23Apparently PyTorch oxidation has started. > Monarch is split into a Python-based frontend, and a backend implemented in Rust. Other than that, looks like a quite interesting project.
Multiple sources say that it is an experimental framework around PyTorch, not a replacement. People will still get to enjoy a circular graph using std::shared_ptr with memory leaks. It's a pity they don't do a complete rewrite with a functional language as the driver.
> It's a pity they don't do a complete rewrite with a functional language as the driver.
It's open source, so seeing such an extension would be quite cool. There's much that could be done with native Rust actors and code that get maybe at what you want, but nothing precludes mixing PyTorch and other backends.
For example, you could wrap a C++ inference engine as part of one of the actors generating data for other actors doing distributed training.
Re: PyTorch Monarch
#24Apparently PyTorch oxidation has started. > Monarch is split into a Python-based frontend, and a backend implemented in Rust. Other than that, looks like a quite interesting project.
This is a new project right? Not the oxidation of an existing one.
Yup, hyperreactor, one of the new crates that's part of it, does some particularly interesting things for efficient parallel distributed channels.
Re: PyTorch Monarch
#25Interesting - this seems to target a different layer than services like Tinker ( https://thinkingmachines.ai/blog/announcing-tinker/ ). Monarch provides the infrastructure primitives while Tinker is a managed finetuning service. Could someone build something like Tinker on top of Monarch?
Yup, there's stuff like https://pytorch.org/blog/introducing-torchforge/ on top of it now
Nice, so the open source equivalent now exists. Meta basically commoditized Tinker's($12B valuation) value prop by giving away the infra (Monarch) and the RL framework (TorchForge). Will be interesting to see how a managed service competes with free + open source at this layer.
Re: PyTorch Monarch
#26"Our Rust-based backend facilitates our performance, scale, and robustness — we amply use Rust’s fearless concurrency in Monarch’s implementation"
Found a few typo's. The em dash makes me suspect an LLM was involved in proofreading
Re: PyTorch Monarch
#27Apparently PyTorch oxidation has started. > Monarch is split into a Python-based frontend, and a backend implemented in Rust. Other than that, looks like a quite interesting project.
Multiple sources say that it is an experimental framework around PyTorch, not a replacement. People will still get to enjoy a circular graph using std::shared_ptr with memory leaks. It's a pity they don't do a complete rewrite with a functional language as the driver.
Interesting, by the way, you can replicate the experience in Rust.
Re: PyTorch Monarch
#28[deleted]
Re: PyTorch Monarch
#29It is a nice project. I have questions.
- Is this similar to openMPI?
- How is a mesh established? Do they need to be on the same host?
Re: PyTorch Monarch
#30"Our Rust-based backend facilitates our performance, scale, and robustness — we amply use Rust’s fearless concurrency in Monarch’s implementation" Found a few typo's. The em dash makes me suspect an LLM was involved in proofreading