351–360 of 462 posts
Earlier quoted context omitted.
People haven't built reliable ternary electronics, though. Soviets tried with Setun, but they eventually had to resort to emulating each trit with two hardware bits (and wasting one state out of the possible four).
If you are are using two bits anyway, you might as well represent (-2, -1, 0, 1) instead of ternary?
Earlier quoted context omitted.
With existing common quantization techniques, a 70b model quantized to 3-bit still drastically outperforms an unquantized 35b model.
Are you sure? I was under impression that 3b quantization still results in a significant degradation. Which quantization method are you talking about?
https://github.com/ggerganov/llama.cpp/pull/1684#issue-17396...
Earlier quoted context omitted.
Yeah, sorry, needed a much bigger canvas than a comment to explain. Let me try again. The example I took was to show mapping from one space to another space and it may have just come across as not learning anything. Yes. You are right it was someone else's pretrained LLM. But this new space learnt the latent representations of the original embedding space. Now, instead of the original embedding space it could also ha…
> how we can draw those planes without the iterative training process. Sounds interesting, but this is the part I would need more explanation on. Just started reading your linked blog, I see it goes into some details there.
Earlier quoted context omitted.
We have been experimenting with the paper( https://www.researchgate.net/publication/372834606_ON_NON-IT... ). There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation. The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. U…
I find this extremely interesting. Do you share the source code of the process? any more references?
The approach is used to solve other problems and papers have been published under https://www.researchgate.net/profile/K-Eswaran
We are currently trying a build a full fledged LLM using just this approach(no LLM training etc) and also an ASR. We should have something to share in a couple of months.
How would you use this in something like PyTorch? There’s no ternary data type.
Earlier quoted context omitted.
Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…
They train using Straight Through Estimator but is cited in the previous BitNet paper. What happen to the TrueNorth Chip? I think investing in specialized hardware for AI is a good bet.
Btw TrueNorth project evolved into "NorthPole" chip by the same group, and was recently in the press. From afar NorthPole looks like an interesting design point and leverages on-chip memory (SRAM)--so it's targeting speed and efficiency at the expense of memory density (so perhaps like Groq in some respects). Tbh I haven't followed the field closely after leaving the group.
E.G. are we just determining the best program by rearranging 1s and 0s?
Earlier quoted context omitted.
Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…
That’s really interesting to see the breadcrumb trail goes back that far. So what are the most important insights in this paper compared to what was previously done? I assume there’s more context to the story and it’s not just that no one thought to apply the concepts to LLM’s until now?
But in fairness, getting these techniques to work at scale is no small feat. In my experience quantization aware training at these low bit depths was always finicky and required a very careful hand. I'd be interested to know if it has become easier to do, now that there are so many more parameters in LLMs.
In any case full kudos to the authors and I'm glad to see people continuing this work.
There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance…