PyTorch Monarch

Apparently PyTorch oxidation has started.

> Monarch is split into a Python-based frontend, and a backend implemented in Rust.

Other than that, looks like a quite interesting project.

I assume this is similar to Ray?

I'm also curious what's the use case of this over Ray. Tighter integration with PyTorch/tensors abstractions?

I made my own single controller PyTorch extension [1], though mines doesn't yet support cross node communication. I found it interesting to compare how Monarch makes things performant. I believe Monarch also uses cloudpickle for code to be shared among all nodes, which is probably the only way to performantly have various nodes execute work as that ends up being a one time setup cost. I found the fanning out of sending messages from the single controller to be really interesting, so the controller is unlikely to be the bottleneck besides any synchronous operations.

As far as things that might be a performance loss here, one thing I'm wondering is if custom kernels are supported. I'm also wondering how much granularity of control there is with communication between different actors calling a function. Overall, I really like this project and hope to see it used over multi-controller setups.

[1] https://github.com/alyxya/mycelya-torch

FB should create a pytorch foundation and set it free before they fuck it up.