[dupe]
Some more discussion a few weeks ago: https://news.ycombinator.com/item?id=40620955
Researchers upend AI status quo by eliminating matrix multiplication in LLMs
11–15 of 15 posts
Re: Researchers upend AI status quo by eliminating matrix multiplication in LLMs
#12Heh, Nvidia may want to take steps to bury this. Will likely be a humongous loss for them if it pans out.
Re: Researchers upend AI status quo by eliminating matrix multiplication in LLMs
#13these quantization are throwing away an advantage of analog computers to handle imprecise "floats"
Re: Researchers upend AI status quo by eliminating matrix multiplication in LLMs
#14The relevant paper: https://arxiv.org/abs/2406.02528 In summary, they forced the model to process data in ternary system and then build a custom FPGA chip to process the data more efficiently. Tested to be "comparable" to small models (3B), theoretically scale to 70B, unknown for SOTAs (>100B params). We have always known custom chips are more efficient especially for tasks like these where it is basically approximat…
With a ternary system, would we expect 1/3 of the elements to be zero? I kind of wonder about using a sparse MM, then they wouldn’t have to represent 0 and the could just use one bit to represent 1 or -1. 66% density is not really very sparse at all though.
Re: Researchers upend AI status quo by eliminating matrix multiplication in LLMs
#15The pre-print is https://doi.org/10.48550/arXiv.2406.02528