After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I don't know what that means for the economics long term, naively way less costs maybe means new entrants without an entire cloud available can compete easier? I do wonder if something like this has already been found and implemented by either OpenAI or Google.
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
21–30 of 462 posts
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#22Sooo, short Nvidia?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#23Sooo, short Nvidia?
Depends if this results in more efficient models or simply larger, more capable models.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#24What does it mean for future hardware if it's not using floating point matrix multiplication units?
https://stackoverflow.com/questions/45373679/why-is-it-faste...
As per answer, the reason float is faster than in is because a) hardware companies provide float ALUs than integer ALUs and b) float FMA is a thing, while integer FMA isn't. Both are because currently most HPC-like loads use floats instead of integers, not because of intrinsic hardware reasons.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#25Can someone versed in the ways of math explain how this is different from previous quantization methods?
And specifically, seeing how going from 16fp to 8bit mostly gives same perplexity while anything further seems to lose quality / dumb down the model, how is this even less precise method is able to achieve this?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#26What does it mean for future hardware if it's not using floating point matrix multiplication units?
Expect Nvidia to advertise with their TOPS numbers instead of their FLOPS.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#27Sooo, short Nvidia?
These still run on GPUs
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#28After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…
It also means the largest models can be scaled up significantly with the same inference budget.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#29After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…
After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher.
In other words, only inference cost is holding it back from completely changing everything.
So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#30Shouldn’t that be “1-trit”?
Read the pdf https://arxiv.org/pdf/2402.17764.pdf they call it 1-bit everywhere.
I don't know why do they do this, 1-bit seems to be a very wrong name for {-1, 0, 1}.