Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

261–270 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#261

Post by mikewarot »

phdelightful wrote: post #250
mikewarot wrote: post #236

Earlier quoted context omitted.

>maybe NAND gates are not the ideal fundamental building block here? It's my long held opinion that LUTs (Look Up Tables) are the basis of computation for the future. I've been pondering this for a long time since George Gilder told us that wasting transistors was the winning strategy. What could be more wasteful than just making a huge grid of LUTs that all interconnect, with NO routing hardware? As time goes by, th…

It’s been a long time since I worked on FPGAs, but it sounds like FPGAs! What do you see as the main differences?

No routing, no fast lines that cut across the chip, which cut way down on latency, but make FPGAs harder to build, and especially hard to compile to once you want to use them.

All that routing hardware, and the special function units featured in many FPGAs are something you have to optimize the usage of, and route to. You end up with using solvers, simulated annealing, etc... instead of a straight compile to binary expressions, and mapping to the grid.

Latency minimization is the key to getting a design to run fast in an FPGA. In a BitGrid, you know the clock speed, you know the latency by just counting the steps in the graph. BitGrid performance is determined by how many answers/second you can get from a given chip. If you had a 1 Ghz rack of BitGrid chips that could run GPT-4, with a latency of 1 mSec per token, you'd think that was horrible, but you could run a million such streams in parallel.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#262

Post by PaulHoule »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I am not startled at all. Dense vector representations are pretty silly, they can’t really be the road to knowledge representation.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#263

Post by spencerchubb »

mise_en_place wrote: post #191
vessenes wrote: post #181

Earlier quoted context omitted.

I'd be VERRY cautious about being excited here. My priors are like this: 1. Initial training of a neural network moves all weights around a large amount at first. 2. Later training of the network adjusts them a small amount. 3. An undertrained network will therefore look a lot like figuring out "positive, negative, or 0?" for each node during early training. If all these things are true, then 1. Early training of an…

Intuitively I've always been a bit skeptical of quantization. Wouldn't there be a tiny loss in precision by doing this type of quantization? I could imagine the error function increasing by utilizing these types of techniques.

Yes each weight will not be able to "learn" as much if it has less bits of precision. But the idea is that you can use more weights, and the big question is whether these low-precision weights can make the model more accurate, as a whole.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#264

Post by p1esk »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Ternary networks have been used since 2015. There are hundreds of papers. They all require full QAT (training from scratch). Not sure why you’re shocked.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#265

Post by acchow »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

Conversely, this also implies our current model sizes can still embed a ton more “understanding”

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#266

Post by fabmilo »

londons_explore wrote: post #47

Powers of 3 don't pack well into binary memory... A 1 bit multiplier in silicon is a single logic gate, but a ternary decoder to decode a packed tri-state 'weight' is bigger. I therefore suspect that this method will be extended to make all weights simple 1 or 0 (ie. Binary). Perhaps that will be done by having half the weights have 1 or 0 values, while the other half are -1 or 0.

can't you have 2 bits ? first bit for the sign second bit for the 1 0 you can represent -1 +1 +0 -0

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#267

Post by cs702 »

gliptic wrote: post #233
cs702 wrote: post #230

Earlier quoted context omitted.

Thank you. Your key point -- that so far all models with the proposed methods may have been only "grossly trained" -- is compelling. If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That seems sensible to me, and makes replication easier, but I agree we need more to see extensive testing, after more extensi…

They also trained 3B with 2 trillion tokens. > The number of training tokens is a crucial factor for LLMs. To test the scalability of BitNet b1.58 in terms of tokens, we trained a BitNet b1.58 model with 2T tokens following the data recipe of StableLM-3B [ TBMR], which is the state-of-the-art open-source 3B model. > [..] > Our findings shows that BitNet b1.58 achieves a superior performance on all end tasks, indicati…

You're right. Thank you for pointing that out!

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#268

Post by ein0p »

ricardobeat wrote: post #243
ein0p wrote: post #168

How is it a 1 bit LLM if 2 bits are required for each weight (and one of the 4 possible states is wasted to be able to represent 0)

As someone else pointed out here, you can store 5 ternary values in 1 byte, 3^5 == 243.

That’s still not 1 bit, and that would basically destroy whatever perf advantage you might hope to get if you want to keep the model in memory in that format rather than unpack it on load.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#269

Post by arunk47 »

Klipper3 wrote: post #51

The theoretical capacity of a binary network is 69% of the capacity of a full-weight network, so it makes sense that LLM would converge to 1-bit networks in the long term. It's nice to finally see practical networks reach the theoretical limits found in the statistical mechanics of Ising models. A good pointer to efficient 1-bit training, from the statistical mechanics point of view, is here: https://www.pnas.org/doi…

What is stopping us right now from doing this one bit networks ?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#270

Post by pennomi »

SushiHippie wrote: post #189
creshal wrote: post #124

Earlier quoted context omitted.

> Could the additional -0 carry some pseudo-gradient information, ("the 0 leaning towards the negative side")? Probably, but is it worth the cost? One of the goals behind BitNet and this paper is to find a way to implement LLMs as efficiently in hardware as possible, and foregoing floating point semantics is a big part of it. I'm not sure if there's a way to encode -0 that doesn't throw out half the performance gains…

But if I understand it correctly, they already need to use 2 bits, one for the sign and another one for the value, so there is already one wasted state, which could be used for -0.

You can pack two trits into three bits, however. So one byte could hold 5 values instead of 4.
Post reply on HN