FB should create a pytorch foundation and set it free before they fuck it up.
PyTorch Monarch
11–20 of 45 posts
Re: PyTorch Monarch
#12> This lets us avoid single-host bottlenecks, effectively using the whole mesh as a distributed cluster for message forwarding. (Cite scalability numbers here.)
In case someone that can fix this is reading here
Re: PyTorch Monarch
#13Re: PyTorch Monarch
#14I assume this is similar to Ray?
I had the same thought, especially because of their recent collaboration.
https://pytorch.org/blog/pytorch-foundation-welcomes-ray-to-...
Re: PyTorch Monarch
#15Apparently PyTorch oxidation has started. > Monarch is split into a Python-based frontend, and a backend implemented in Rust. Other than that, looks like a quite interesting project.
This is a new project right? Not the oxidation of an existing one.
Re: PyTorch Monarch
#16Interesting - this seems to target a different layer than services like Tinker (https://thinkingmachines.ai/blog/announcing-tinker/). Monarch provides the infrastructure primitives while Tinker is a managed finetuning service. Could someone build something like Tinker on top of Monarch?
Re: PyTorch Monarch
#17Interesting - this seems to target a different layer than services like Tinker ( https://thinkingmachines.ai/blog/announcing-tinker/ ). Monarch provides the infrastructure primitives while Tinker is a managed finetuning service. Could someone build something like Tinker on top of Monarch?
Yup, there's stuff like https://pytorch.org/blog/introducing-torchforge/ on top of it now
Re: PyTorch Monarch
#18Apparently PyTorch oxidation has started. > Monarch is split into a Python-based frontend, and a backend implemented in Rust. Other than that, looks like a quite interesting project.
Multiple sources say that it is an experimental framework around PyTorch, not a replacement. People will still get to enjoy a circular graph using std::shared_ptr with memory leaks.
It's a pity they don't do a complete rewrite with a functional language as the driver.
Re: PyTorch Monarch
#19I assume this is similar to Ray?
I'm also curious what's the use case of this over Ray. Tighter integration with PyTorch/tensors abstractions?
That.
Also, it has RDMA. Last I checked, Ray did not support RDMA.
There are probably other differences as well, but the lack of RDMA immediately splits the world into things you can do with ray and things you cannot do with ray
Re: PyTorch Monarch
#20This seems strictly less powerful than Jax, which comes with a powerful compiler that optimises how cross-node communication is conducted.
Nah, focusing on a different controller paradigm. Jax is focused on multi-controller SPMD, while this is focused on a single-controller setup. Both have their place, with single-controller being generally easier to reason about, and multi-controller more optimal for certain dataflows. There's also some interesting mixes of the two control paradigms.