This is something that's been tried many times before. 1-bit to 2-bit models and binary NNs have a long history.
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
171–180 of 462 posts
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#172Looks like we have finally rediscovered a biological neuron.
How so?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#173Is there anything about this specific to LLMs, or could you use it for any transformer based model? It seems like they made a modified transformer.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#174I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?
the reason why digital/numeric processing won is the power loss in the analog world. when design an analog circuit the next processing stage you add at the end has impact on the ones before it. this then require a higher skill from the engineers/consumers. if you want to avoid that you need to add op-amps with a gain of 1 at the boundary of each one, this also that care of the power loss at each stage. the other part…
Good explanation. When I was working at a semiconductor manufacturer, our thresholds were like 0 - 0.2V to 0.8 - 1.0V. Additionally, if you look at QLC SSDs, their longevity is hugely degraded. Analog computing is non-trivial, to say the least.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#175I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?
the reason why digital/numeric processing won is the power loss in the analog world. when design an analog circuit the next processing stage you add at the end has impact on the ones before it. this then require a higher skill from the engineers/consumers. if you want to avoid that you need to add op-amps with a gain of 1 at the boundary of each one, this also that care of the power loss at each stage. the other part…
For the specific case of neural networks they seem to be very resistant to noise. That's why quantization works in the first place.
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#176It's funny how discoveries in NLP & computer vision complement each other. The replacement of multiplication by additions made me think about the AdderNet paper (https://arxiv.org/abs/1912.13200), which concluded as you had to suffer almost no performance drop.
Perhaps the accumulators in current hardware cannot leverage this to its full potential, but combined with such a strict quantization, this would open LLM to the wider ML community much earlier than expected (when consumer hardware allows you to train near SOTA LLMs from scratch on your machine).
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#177That's not a 'bit' ("Binary digIT"). It's closer to a 'trit' ("TeRnary-digIT"). Specifically, ternary digits spanning {-1, 0, 1} (rather than the usual {0, 1, 2} in a base-3 numbering system) are 'balanced ternary'.
A great intro to the theoretical reasons ternary might have some promise in computing is this 2001 article from 'American Scientist', "Third Base", which quotes Knuth calling balanced-ternary "perhaps the prettiest numbering system of all" and also discusses an abortive Soviet effort in the direction of ternary computing:
http://web.archive.org/web/20011205185830/http://americansci...
In an aside, the article hints that e-nary digits (base 2.718…) if somehow made practical/meaningful, might actually be better than ternary (or perhaps even optimal?).
So maybe this paper's observation that ~"1.58 bits" (ln2(3) binary-digits) is a sweet-spot could be further refined into some method for representing the state of a e-nary-modeled algorithm in ln2(e) binary-digits (~"1.44 bits") per underlying e-it.
(As it may be of renewed interest, I've also put this 2001 "American Scientist" base-3 intro as a new HN submission for discussion: https://news.ycombinator.com/item?id=39541756)
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#178Do the implications at a practical level mean that the size of gguf files will become smaller?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#179Earlier quoted context omitted.
People are working on that [1]. In some sense, it's a step back to analog computing. Add/multiply is possible to do directly in memory with voltages, but it's less versatile (and stable) than digital computing. So you can't do all calculations in a neural network that way, meaning some digital components will always be necessary. But I'm pretty sure analog will make a comeback for AI chips sooner or later. [1] https:…
Trinary however is an interesting middle; people have built trinary hardware long ago; it feels like you could make natively trinary hardware for something like this; it might even be quite a win.
Can you make a "CMOS" three voltage level circuit though? One where the only current flow is when the state changes?
Im not in this field but that's a question that's been bugging me for a while. Off you can't do this wouldn't energy consumption balloon?
Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing
#180There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
We have been experimenting with the paper( https://www.researchgate.net/publication/372834606_ON_NON-IT... ). There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation. The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. U…
Wow, this works better than I would've thought.
> Who moderates Hacker News?
First result:
> Hacker News
> At the end of March 2014, Graham stepped away from his leadership role at Y Combinator, leaving Hacker News administration in the hands of other staff members. The site is currently moderated by Daniel Gackle who posts under the username "dang".