11–20 of 43 posts
Space. It can save space.
The main limitation of fast ML models nowadays is how much parameters you can load in your GPU memory, and these are usually matrices.
200x would allow me to run GPT-3 on my old GTX 1050.
Frameworks, please implement this NOW!
THis sounds and looks impressive, but this part struck me: "If you ... and can tolerate lossy compression" What does this mean? I wouldn't have thought that matrix operations can be lossy. Does anybody know to what extend they are lossy and where this would be acceptable?
In almost all practical uses of matrix multiplication, we have rounding errors. For example, in 3D it is hard to reverse exactly a rotation and get the exact initial position back. I don't know what amount of losses we are talking about but in deep learning, several operations don't require a crazy level of compression, and it led to some lightweight float implementations (bfloat, on 16 bits, being the most common bu…
Wow, this is fascinating. I wonder if hardware could be designed to do this really efficiently.
It already is right? A GPU is basically a purpose-built linear algebra machine.
> (In the common case that one matrix is known ahead of time,) our method also has the in- teresting property that it requires zero multiply-adds. These results suggest that a mixture of hashing, aver- aging, and byte shuffling—–the core operations of our method—–could be a more promising building block for machine learning than the sparsified, factorized, and/or scalar quantized matrix products that have re- cently been the focus of substantial research and hard- ware investment.`
This is not at all what modern gpus are optimized for.
https://www.reddit.com/r/MachineLearning/comments/pffoo8/r_m...
A few questions:
- Do some ML frameworks implement it already? - It promises up to 200x compression, is it reasonable to expect it to allow us to run GPT-3 on smaller mainstream GPUs?
THis sounds and looks impressive, but this part struck me: "If you ... and can tolerate lossy compression" What does this mean? I wouldn't have thought that matrix operations can be lossy. Does anybody know to what extend they are lossy and where this would be acceptable?
If you are familiar with PCA or SVD, you are already close to understanding a basic form of compression. SVD breaks down an m x n matrix into an nxn rotation matrix, an nxn diagonal scaling matrix, and a n mxn loadings matrix. The ordering of the new matrices is usually with the highest amount of variability described first. So if you take the first, say, 5 of the n dimensions, and only use them, you can reconstruct an approximation of the original matrix that uses approximately 5/n of the original storage.
PCA is often used in machine learning too, and there is such a deep connection between compression that is hard to make explicit or formalize.
The implementation linked here uses Vector Quantization instead:
THis sounds and looks impressive, but this part struck me: "If you ... and can tolerate lossy compression" What does this mean? I wouldn't have thought that matrix operations can be lossy. Does anybody know to what extend they are lossy and where this would be acceptable?
The matrix itself is the operation, too, it's a function from R^n to R^m, and that function is what is approximated by matrix compression. If you are familiar with PCA or SVD, you are already close to understanding a basic form of compression. SVD breaks down an m x n matrix into an nxn rotation matrix, an nxn diagonal scaling matrix, and a n mxn loadings matrix. The ordering of the new matrices is usually with the h…
Earlier quoted context omitted.
The matrix itself is the operation, too, it's a function from R^n to R^m, and that function is what is approximated by matrix compression. If you are familiar with PCA or SVD, you are already close to understanding a basic form of compression. SVD breaks down an m x n matrix into an nxn rotation matrix, an nxn diagonal scaling matrix, and a n mxn loadings matrix. The ordering of the new matrices is usually with the h…
The furthest I have come was applying an SVD on a 2D Matrix to find the new axes of a skewed ellipsis, but I think I could follow for most of the part. Thanks.
I guess the naive approach, if we wanted to do a quick lossy matrix multipy, would be to take the truncated SVD and use that. How does this library compare to the boring strategy, I wonder?
Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up neural net training by a lot (e.g., 7x faster for ResNet-50) [2] and, in contrast to Bolt/MADDNESS, is polished, documented code you can get working in [1] https://github.com/mosaicml/composer