Seems to require Tensorflow. That makes it basically irrelevant?
PyTorch too?[1]
[1] https://docs.ray.io/en/latest/train/api/doc/ray.train.torch....
ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
11–17 of 17 posts
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#12Yes but do they pay Anyscale anything for this use of Ray?
Founder of Anyscale is the Berkeley student who helped start Ray and is usually happy to give free advice. Anyscale is to Ray as Databricks is to Spark.
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#13Yes but do they pay Anyscale anything for this use of Ray?
This is all open source Ray. None of the AWS benchmarks were run on Anyscale platform. You can install and run OSS Ray on any of the popular clouds, including your laptop.
All the scripts for you to run in your OSS Ray cluster are here: https://github.com/ThirdAILabs/Public-Benchmarks/blob/main/c...
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#14Very cool tech! I remember seeing the SLIDE paper and then being shocked at the lack of high profile follow-up work.
Is there any plan to open source BOLT? This would be extremely valuable to the community and in reducing Nvidia's chokehold.
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#15Very cool tech! I remember seeing the SLIDE paper and then being shocked at the lack of high profile follow-up work. Is there any plan to open source BOLT? This would be extremely valuable to the community and in reducing Nvidia's chokehold.
Thanks for reading! This is a great question. While we currently don't have immediate plans to open source BOLT, we are still in the very early stages of the company and may consider it down the road. Although BOLT is currently closed-source, we do offer free trial licenses https://www.thirdai.com/try-bolt/
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#16Their methods rely on heuristics to sparsify NNs. They compare their sparse and dense methods to PyTorch and Tensorflow and get worse perf on dense inferences but better on sparse. It would be useful to know if PyTorch runs faster or slower than their methods using their sparsified matrix techniques.
[dead]
Re: ThirdAI Uses Ray for Parallel Training of Billion-Parameter NN on Commodity CPUs
#17Seems to require Tensorflow. That makes it basically irrelevant?
PyTorch too?[1] [1] https://docs.ray.io/en/latest/train/api/doc/ray.train.torch....
Ray provides integrations for PyTorch and a variety of other frameworks, but doesn't depend on them (unless you're using those integrations of course). The core library is just general purpose distributed compute.