381–390 of 462 posts
Earlier quoted context omitted.
Trinary however is an interesting middle; people have built trinary hardware long ago; it feels like you could make natively trinary hardware for something like this; it might even be quite a win.
Can you make a "CMOS" three voltage level circuit though? One where the only current flow is when the state changes? Im not in this field but that's a question that's been bugging me for a while. Off you can't do this wouldn't energy consumption balloon?
I was reading Exposing Floating Point today (as Airfoil is on the HN front page and I was perusing the archive of the author). It's a blog explaining the inner workings of floating point representations. About zero values it says [0]: > Yes, the floating point standard specifies both +0.0 and −0.0. This concept is actually useful because it tells us from which “direction” the 0 was approached as a result of storing v…
Interesting, how do you use -0 in the add, then? Is -0+1-1 a 0 or a -0? > Could the additional -0 carry some pseudo-gradient information It looks like training was done on fp32 or bf16. Low-bit quantization is approximated with STE during training. I'd expect training itself cause each point to "polarize" towards 1 or -1. > 2-bit quantizations being proposed Symmetric (i.e. without 0) exponential values were pretty p…
Earlier quoted context omitted.
I find this extremely interesting. Do you share the source code of the process? any more references?
Unfortunately the source code is currently not open sourced. Some more details at ( https://www.researchgate.net/publication/370980395_A_NEURAL_... ), the source code is built on top of this. The approach is used to solve other problems and papers have been published under https://www.researchgate.net/profile/K-Eswaran We are currently trying a build a full fledged LLM using just this approach(no LLM training etc) an…
It says here ( https://www.researchgate.net/publication/370980395_A_NEURAL_... ) that each layer can be represented as a matrix multiplication (equation 3): Ax = s
So concatenating multiple layers could just be reduced to a single matrix multiplication?
If there is no non-linearity I don't see how this could replace neural networks, or am I missing something?
Earlier quoted context omitted.
If this dethrones Nvidia, it would be a wonderful side effect
It's more likely that Nvidia will offer support to INT2 in the next generation and keep their dominance.
Intel and AMD could also implement support in their "next generation" and that would be huge.
There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…
> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…
No, unless I'm mistaken it's a huge impact: it means the matrix product is separable: basically, it's a O(n²) algorithm, and not O(n3): add together all the c_j = sum(a_i_j), d_i = sum(b_i_j), and the final results are all the combinations of cj+di. And even then, half that is unnecessary because the d_i can all be pre-computed when before inference since they are weights.
But I skimmed over the paper, and didn't found the part where it was explained how they replace the product by additions: from what I understand, they remplace multiplications by bi by selecting +ai, 0, or -ai. So the final matrix multiplication can be implemented by only additions, but only because the weights are 1,0,-1 they avoid multiplications altogether. This is really different from what the GP said (remplacing a0*b0+... by a0+b0+...).
These models will are compatible with llama.cpp out of the box, we (GigaML - https://gigaml.com ) are planning to train a small model (3-4B, 1-bit, opensource) with the latest stack-v2 dataset released today. Let me know if anyone is interested in collaborating with us.
I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?
Earlier quoted context omitted.
Trinary however is an interesting middle; people have built trinary hardware long ago; it feels like you could make natively trinary hardware for something like this; it might even be quite a win.
Can you make a "CMOS" three voltage level circuit though? One where the only current flow is when the state changes? Im not in this field but that's a question that's been bugging me for a while. Off you can't do this wouldn't energy consumption balloon?
The mathematics of the BNNs are sound. The shannon entropy of a word is really small (I vaguely remember ~2 bits). Also all neural networks are ridiculously over provisioned. I worked on 7 years ago trying to efficiently binarize CNNs from existing models. It the difficult was getting training running without the losses going to high. I think that vision models will be much more difficult to binarize, but you might n…
Earlier quoted context omitted.
Not true anymore, but it also highly depends on what your definition of "a good one" is. Many people find Mistral 7B to be excellent, around gpt-3.5 level of good. Mistral 7B normally requires like 20gb VRAM, but with llama.cpp and quantization, you could even run it on your phone (albeit bad quality). Quantization >= q4_K_M seem to provide nearly as good responses as the unquantized model, and q4_K_M only needs ~7GB…
Oh Jesus so basically it’s very feasible for me to run my own local llm on a NAS or a server or something… well I guess it’s time for me to get on with the times… Thanks!