Viewing profile — ffast-math

I named it this in 2017 and was only worried about name collisions with other GitHub repos and ML algorithms. Also it's a backronym for Based On Lookup Tables + sounds at least som…

Nope. I'd love to though.

This master's thesis sort of does it for individual layers, but it doesn't have any fine-tuning yet so it completely wrecks the accuracy: https://github.com/joennlae/halutmatmul . …

No ML frameworks implement it yet, though I'd be happy to work with people from the PyTorch/TF/JAX/CUDNN/CUTLASS/etc. teams (or volunteers) if anyone wants to make this happen. Als…

IMO it would be super cool and I hope someone does it. There are a lot of interesting tradeoffs around which techniques to use for which matrix sizes and under which assumptions ab…

email. @mosaicml.com

Thanks for posting it! It should be possible to get large speedups on CPUs, but the trick will be gradually approximating each of the layers in the model (see my reply to sibling c…

Yes. It's another research project to make this happen, but I think it would be fairly straightforward. The issue is that you can't backprop through the assignment step, so you get…

We found sparse, truncated PCA to be the most competitive baseline. We beat it by a lot (see the paper [1]), but the other big drawback is that trading off the rank vs sparsity was…

Exactly. You can run it on sparse inputs. It's just that our implementation doesn't exploit the sparsity, so we don't claim that it will work better.

Definitely. On CPUs, you could make this 2x faster pretty easily with just another execution port for vpshufb / vtbl and a 4bit lo and hi unpack instruction. Though the real speedu…

There's definitely a tradeoff between speed and accuracy. We characterize this for various problems in the paper ( https://arxiv.org/pdf/2106.10860.pdf ), but tl;dr is that it spee…

Author here. Ask me anything--happy to answer questions. Also, if you like this kind of work, you might like what I've been building for the past year: Composer [1]. It speeds up n…

I think we only claim to be able to preprocess a matrix at "up to" 100GB/s/core. The overall matrix product will take longer and depend on the matrix shapes. To simplify Section 1.…

We have a generalization guarantee in Section 4.5. It's not especially tight though; in practice, the errors from different codebooks tend to be mostly independent, and you get nic…

Almost certainly not, sadly, unless maybe there's a ton of correlation in the input and you can tolerate quite a bit of error.

So this misses a few aspects of why the method works: - You can't actually get a speedup from the proposed approach. You'd need a lookup table of size b^2 to multiply two b-bit num…

I'm actually not quite sure what you mean by breaking down into two-dimensional operations. We use operations on pairs of vectors, but nothing is assumed to be two-dimensional, and…

You could still optimize the prototypes, so fine-tuning with this in place would be possible (see, e.g., [1]). But we don't yet have data on how well this would work using our exac…

Yes, basically correct. A couple notes/clarifications for other readers: - The rows a of A are "random," but in the sense of being drawn from some distribution for which we have a …

We compared to several frequent directions variants, Fast Johnson–Lindenstrauss, some other hashing-based methods, and a bunch of other approximate matrix multiplication approaches…

There's been some work on doing adds instead of multiplies (e.g., https://arxiv.org/abs/2012.03458 ). And I think float8 will roughly be doing this under the hood. Personally, I'm …

Great observations. I see this paper as the first in a three-part series. The second part is specializing it for convolution (which has additional structure to exploit), and the th…