any thoughts on trying to build a sort of vq-blas?
Bolt: Faster matrix and vector operations that run on compressed data
31–40 of 43 posts
Re: Bolt: Faster matrix and vector operations that run on compressed data
#32Earlier quoted context omitted.
Thanks for posting it! It should be possible to get large speedups on CPUs, but the trick will be gradually approximating each of the layers in the model (see my reply to sibling comment). It's not conceptually difficult, but will require a fair amount of C++ work to port the code to GPUs* for training; and it will probably go slower than dense ops on modern GPUs due to tensor cores not supporting our memory layout.…
Thank you, I will try to take this up. What would be the best way to reach out to you?
email. @mosaicml.com
Re: Bolt: Faster matrix and vector operations that run on compressed data
#33any thoughts on trying to build a sort of vq-blas?
IMO it would be super cool and I hope someone does it. There are a lot of interesting tradeoffs around which techniques to use for which matrix sizes and under which assumptions about read vs write ratios, what you have a training set for, whether you can fuse compression intro previous ops, etc.
Re: Bolt: Faster matrix and vector operations that run on compressed data
#34This is actually from a paper published last year: https://www.reddit.com/r/MachineLearning/comments/pffoo8/r_m... A few questions: - Do some ML frameworks implement it already? - It promises up to 200x compression, is it reasonable to expect it to allow us to run GPT-3 on smaller mainstream GPUs?
No ML frameworks implement it yet, though I'd be happy to work with people from the PyTorch/TF/JAX/CUDNN/CUTLASS/etc. teams (or volunteers) if anyone wants to make this happen.
Also, while you can get 200x compression, I do want to emphasize that there's a speed vs quality tradeoff and the results will vary by problem. We have much more careful statements in the paper about the exact problem setup, tradeoffs, etc. Also, as I've mentioned in other comments, it probably won't help too much on modern GPUs due to their acceleration of dense GEMMs but not shuffles. CPU inference is the killer app here.
Re: Bolt: Faster matrix and vector operations that run on compressed data
#35Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet-50) [2] and, in contrast to Bolt/MADDNESS, is polished, documented code you can get working in [1] https://github.com/mosaicml/composer [2] https://www.mosaicml.com/blog/mosaic-resnet
Are you aware of any code/projects that convert something like a fully trained resnet50 model into a maddness optimized, approximate model?
An approximate but speedier resnet inference model that runs on a CPU would be useful even if it’s not quite as fast/accurate as a GPU inference model, since currently the cost to run a GPU is typically higher than CPUs.
Re: Bolt: Faster matrix and vector operations that run on compressed data
#36Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet-50) [2] and, in contrast to Bolt/MADDNESS, is polished, documented code you can get working in [1] https://github.com/mosaicml/composer [2] https://www.mosaicml.com/blog/mosaic-resnet
Are you aware of any code/projects that convert something like a fully trained resnet50 model into a maddness optimized, approximate model? An approximate but speedier resnet inference model that runs on a CPU would be useful even if it’s not quite as fast/accurate as a GPU inference model, since currently the cost to run a GPU is typically higher than CPUs.
This master's thesis sort of does it for individual layers, but it doesn't have any fine-tuning yet so it completely wrecks the accuracy: https://github.com/joennlae/halutmatmul.
If someone worked on contributing this functionality to Composer [1] I'd be down to help out. I can't justify building it all on my own right now since we're 100% focused on training speedup, but I could definitely meet and talk through it, help code tricky parts, review PRs, etc.
Re: Bolt: Faster matrix and vector operations that run on compressed data
#37Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet-50) [2] and, in contrast to Bolt/MADDNESS, is polished, documented code you can get working in [1] https://github.com/mosaicml/composer [2] https://www.mosaicml.com/blog/mosaic-resnet
Have you done any presentations at the big companies yet?
Re: Bolt: Faster matrix and vector operations that run on compressed data
#38Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet-50) [2] and, in contrast to Bolt/MADDNESS, is polished, documented code you can get working in [1] https://github.com/mosaicml/composer [2] https://www.mosaicml.com/blog/mosaic-resnet
Have you done any presentations at the big companies yet?
Nope. I'd love to though.
Re: Bolt: Faster matrix and vector operations that run on compressed data
#39Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet-50) [2] and, in contrast to Bolt/MADDNESS, is polished, documented code you can get working in [1] https://github.com/mosaicml/composer [2] https://www.mosaicml.com/blog/mosaic-resnet
Are you aware of any code/projects that convert something like a fully trained resnet50 model into a maddness optimized, approximate model? An approximate but speedier resnet inference model that runs on a CPU would be useful even if it’s not quite as fast/accurate as a GPU inference model, since currently the cost to run a GPU is typically higher than CPUs.
There's lots of work in this area.
Quantization is a common technique. See for example https://pytorch.org/docs/stable/quantization.html
Re: Bolt: Faster matrix and vector operations that run on compressed data
#40Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet-50) [2] and, in contrast to Bolt/MADDNESS, is polished, documented code you can get working in [1] https://github.com/mosaicml/composer [2] https://www.mosaicml.com/blog/mosaic-resnet
Are you aware of any code/projects that convert something like a fully trained resnet50 model into a maddness optimized, approximate model? An approximate but speedier resnet inference model that runs on a CPU would be useful even if it’s not quite as fast/accurate as a GPU inference model, since currently the cost to run a GPU is typically higher than CPUs.
Not to distract from the clearly good technical work you've done here, but why name it Bolt when there's a Fintech startup with the same name. Among other brand confusion issues (Like the Chevy Bolt).