ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
1–10 of 17 posts
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#2Yes but do they pay Anyscale anything for this use of Ray?
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#3Their methods rely on heuristics to sparsify NNs. They compare their sparse and dense methods to PyTorch and Tensorflow and get worse perf on dense inferences but better on sparse. It would be useful to know if PyTorch runs faster or slower than their methods using their sparsified matrix techniques.
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#4Their methods rely on heuristics to sparsify NNs. They compare their sparse and dense methods to PyTorch and Tensorflow and get worse perf on dense inferences but better on sparse. It would be useful to know if PyTorch runs faster or slower than their methods using their sparsified matrix techniques.
> It would be useful to know if PyTorch runs faster or slower than their methods using their sparsified matrix techniques.
Probably not an easy feat. Sparse support in pytorch is minimal.
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#5imo this sounds like the right move, relying on a single company to produce all the gpus results in massive competition to get them, and a large barrier to entry to buy them
by skipping it you can play a little slower but probably a lot cheaper
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#6Yes but do they pay Anyscale anything for this use of Ray?
you can set up a self-hosted Ray infra to achieve this.
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#7Yes but do they pay Anyscale anything for this use of Ray?
IIUC Anyscale is managed Ray, and also works on Ray full-time. However Ray is open source and you can run it yourself. It's fairly easy to get started with, too.
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#8Seems to require Tensorflow. That makes it basically irrelevant?
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#9Seems to require Tensorflow. That makes it basically irrelevant?
Hi, I am one of the co-authors of the blog from ThirdAI. Our software has no dependency on TensorFlow or any other deep learning framework. You can see the libraries we import for training and try it out for yourself in our public github repo: https://github.com/ThirdAILabs/Public-Benchmarks/blob/main/c...
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#10Seems to require Tensorflow. That makes it basically irrelevant?
Ray does not require Tensorflow, tho the cited workflow might use tf.
Here’s a random example of pytorch on ray: https://www.ray.io/ray-sgd