51–60 of 462 posts
It's nice to finally see practical networks reach the theoretical limits found in the statistical mechanics of Ising models. A good pointer to efficient 1-bit training, from the statistical mechanics point of view, is here:
I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?
So are there any details on the algorithms they used for backprop? I'm not seeing any in the paper other than "we used a lot of tokens".
Does this help? https://arxiv.org/abs/2310.11453 It seems to have more details (it's the paper before the linked one) about the actual training, but I'm scanning it and this isn't my field so maybe it's too light also.
But I imagine they are using the same thing since a bunch of the authors are the same.
Earlier quoted context omitted.
As per answer, the reason float is faster than in is because a) hardware companies provide float ALUs than integer ALUs and b) float FMA is a thing, while integer FMA isn't. Both are because currently most HPC-like loads use floats instead of integers, not because of intrinsic hardware reasons.
If it's desired integer performance could far exceed float performance, since ALUs need less die area than FPUs. If this paper holds, I'd expect that's where custom accelerators will be heading.
edit: also this might be implementable purely using bitwise vector operations. Would need to check the throughput of those.
I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?
Sooo, short Nvidia?
Depends if this results in more efficient models or simply larger, more capable models.
Of course it should be fairly simple for Nvidia to add special silicon and instructions for two-bit addition to a future generation of their cards. But it'll take a while because they already have a roadmap and preexisting commitments. And any competitor doesn't have to copy everything Nvidia does to make floating point numbers go fast, they can just focus on making two-bit data handling and addition go fast.
Too bad there seem to be no pretrained models to download. This is not a quantization method to apply on existing models, so having the pretrained weights is needed if one wants to test it.
It seems that it may be published on GitHub [1] according to HuggingFace [2].
After reading the results I skipped back to the comment section to ask if this was real because it looks a little too good to be true, but figured I should check authors and it's Microsoft research and UCAS so yeah, real. This is going to change a lot of things, obviously the edge computing applications they point out, but also this is going to bottom out the cost of providing high-performance LLMs in the cloud. I do…
After playing with OpenAI's GPT4 API, I'm quite convinced that LLMs would be in everything and everywhere today if inference cost is as low as loading a website and context size is 100x higher. In other words, only inference cost is holding it back from completely changing everything. So if we have a shortcut to getting something like GPT4 to run locally on a small device, watch out.