Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

131–140 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#131

Post by kurisufag »

StableAlkyne wrote: post #116
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

It would be something of a full circle I feel went back to dedicated circuits for NNs - that's how they began life when Rosenblatt built his Perceptron. I remember reading a review on the history in grad school (can't remember the paper) where the author stated that one of the initial interests in NNs by the military was their distributed nature. Even back then, people realized you could remove a neuron or break a co…

>von Braun machine

von neumann? though it is funny to imagine von braun inventing computer architecture as a side hustle to inventing rocket science.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#132

Post by rhaps0dy »

cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I think you need more evidence than this paper (which is very short and light on actual numbers) to be this shocked.

For example, most of the plots in the paper are actually of throughput, memory, etc. all performance characteristics that are better on the ternary version. Which, of course.

The only thing that contains perplexities are Table 1 and 2. There, they compare "BitNet b1.58 to our reproduced FP16 LLaMA LLM in various sizes" on the RedPajama data set. The first thing to note is the perplexities are very high: they're all at least ~9.9, which compared for example with quantized Llama on wikitext-2 which is 6.15 (https://www.xzh.me/2023/09/a-perplexity-benchmark-of-llamacp...). Maybe RedPajama is a lot harder than wikitext-2, but that's a big gap.

I think probably their benchmark (their "reproduced FP16 LLaMA LLM") is just not very good. They didn't invest much in training their baseline and so they handily beat it.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#133

Post by w-m »

rfoo wrote: post #123
w-m wrote: post #119

I was reading Exposing Floating Point today (as Airfoil is on the HN front page and I was perusing the archive of the author). It's a blog explaining the inner workings of floating point representations. About zero values it says [0]: > Yes, the floating point standard specifies both +0.0 and −0.0. This concept is actually useful because it tells us from which “direction” the 0 was approached as a result of storing v…

Interesting, how do you use -0 in the add, then? Is -0+1-1 a 0 or a -0? > Could the additional -0 carry some pseudo-gradient information It looks like training was done on fp32 or bf16. Low-bit quantization is approximated with STE during training. I'd expect training itself cause each point to "polarize" towards 1 or -1. > 2-bit quantizations being proposed Symmetric (i.e. without 0) exponential values were pretty p…

> how do you use -0 in the add

In my mind the two zero values would represent a tiny epsilon around 0, let's say -0.01 and +0.01. Looking at them like this, it would mean

  +0 +0 -0 = +0
  +0 -0 -0 = -0
  +1 * +0 = +0
  -1 * +0 = -0
Performing addition with the same sign count in each group would be problematic. How to decide on the sign of +0-0 or +1-1, other than flipping a coin?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#134

Post by vidarh »

dns_snek wrote: post #120
vidarh wrote: post #117

Earlier quoted context omitted.

Last I checked (and confirmed by repeating it just now) GPT-4 did just fine at adding 2 numbers up, because it knows better now than to do that manually and will express it as Python. It does worse if you try to force it to do it step by step like a child and don't reinforce adherence to the rules every step, because just like humans it gets "sloppy" when you try to get it to repeat the same steps over and over. If y…

The claim was that "LLMs can do math". Below they linked a model from Google that might be capable of that, but as a general rule (and with OpenAI's models specifically) LLMs can't "do math" by any reasonable definition.

I've had it do plenty of math. Some it does badly at, some it does fine. Generally it's not "disciplined" enough to do things that requires lots of rote repetitive tasks, but neither are most humans, and that has improved drastically as they've adjusted it to instead do what most humans do and use tools. Would it be nice if it also got more willing to "stick to it" when given rote tasks? Sure.

But whether or not it can "do maths" to your definition depends very much on what you want it to do, and how you define "do maths". To me it's irrelevant if it's doing the low-level calculations as long as it knows how to express them as code. If I wanted a calculator I'd use a calculator. And I don't consider a calculator able to "do math" just because it can precisely add numbers.

Meanwhile I've had lengthy discussions with GPT about subjects like orbital mechanics and calculating atmospheric effects where it correctly used maths that I had to double-check not because I didn't trust GPT (though I also want't to verify for that reason) but because I didn't know the maths (not that it was anything particularly advanced, but I lost interest in maths during my CS degree and picked the minimum amount of maths I could get away with).

By my definition it can "do maths" just fine. I guess you don't consider my view of that "reasonable". I can live with that, as meanwhile, it will keep doing maths for me when I need it.

Of course this was also a case of moving the goalposts to set up a strawman - in the comment of yours I replied to, you claimed it couldn't reliably add two numbers.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#135

Post by cs702 »

rhaps0dy wrote: post #132
cs702 wrote: post #100

There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise additions (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the pa…

I think you need more evidence than this paper (which is very short and light on actual numbers) to be this shocked. For example, most of the plots in the paper are actually of throughput, memory, etc. all performance characteristics that are better on the ternary version. Which, of course. The only thing that contains perplexities are Table 1 and 2. There, they compare "BitNet b1.58 to our reproduced FP16 LLaMA LLM…

Thank you. I think the paper as it is provides enough evidence to support the claims. If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That's sensible. It allows for easier replication of the results. Otherwise, I agree with you that more extensive testing, after more extensive pretraining, is still necessary.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#136

Post by StableAlkyne »

kurisufag wrote: post #131
StableAlkyne wrote: post #116

Earlier quoted context omitted.

It would be something of a full circle I feel went back to dedicated circuits for NNs - that's how they began life when Rosenblatt built his Perceptron. I remember reading a review on the history in grad school (can't remember the paper) where the author stated that one of the initial interests in NNs by the military was their distributed nature. Even back then, people realized you could remove a neuron or break a co…

>von Braun machine von neumann? though it is funny to imagine von braun inventing computer architecture as a side hustle to inventing rocket science.

Oh fuck, thanks for catching that!

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#137

Post by anon373839 »

btbuildem wrote: post #68
anon373839 wrote: post #5

> BitNet b1.58 can match the performance of the full precision baseline starting from a 3B size. ... This demonstrates that BitNet b1.58 is a Pareto improvement over the state-of-the-art LLM models. > BitNet b1.58 is enabling a new scaling law with respect to model performance and inference cost. As a reference, we can have the following equivalence between different model sizes in 1.58-bit and 16-bit based on the re…

Discussion on HF [1] implies that no, conversion is not helpful. It would take training the model from scratch. 1: https://huggingface.co/papers/2402.17764

It’s a pity if realizing these gains absolutely requires full pre-training from scratch. I imagine more than a few people will at least try to find a way to repurpose the knowledge contained in existing models.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#138

Post by creshal »

ynniv wrote: post #82
sigmoid10 wrote: post #50

Earlier quoted context omitted.

I'd be very surprised if Apple can put something on the level of GPT4 on a handheld. Remember, GPT4 is estimated to be around 1.7 trillion parameters. That's 3.4TB at 16 bit and it would still be ~340GB at 1.58bits. The best we can hope for is a low-ish level few billion parameter model. Which would still be cool on a phone, but as of today these models are nowhere near GPT4.

You don't need "GPT4" though. Mixtral 8x7B is robust and can be run in 36 Gb, 24 Gb if you're willing to compromise. A 1.5 bit quantization should bring it down to 16. That's still a lot compared to the iPhone 15's 6, but it's close enough to imagine it happening soon. With some kind of streaming-from-flash architecture you might be in the realm already.

> With some kind of streaming-from-flash architecture you might be in the realm already.

I thought mmap'ing models to only keep the currently needed pieces in RAM was something that was figured out ~6 months ago? Performance wasn't terribly great iirc, but with how much faster 1.58B is, it should still be okay-ish.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#139

Post by jerf »

cs702 wrote: post #128
creshal wrote: post #118

Earlier quoted context omitted.

> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…

> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even need lower floating-point precision. We don't need precision at all . We only need three symbols to represent every value. > I feel like this follows naturally from having only ternary values,…

An integer is just a concatenation of bits. Floating point appears more complicated but from an information theory perspective it is also just a concatenation of bits. If, for the sake of argument, one replaced a 64-bit int with 64 individual bits, that's really the same amount of information and a structure could hypothetically then either choose to recreate the original 64-bit int, or use the 64-bits more efficiently by choosing from the much larger set of possibilities of ways to use such resources.

Trits are helpful for neural nets, though, since they really love signs and they need a 0.

So from the perspective that it's all just bits in the end the only thing that is interesting is how useful it is to arrange those bits into trits for this particular algorithm, and that the algorithm seems to be able to use things more effectively that way than with raw bits.

This may seem an absolutely bizarre zigzag, but I am reminded of Busy Beavers, because of the way they take very the very small primitives of a Turing Machine, break it down to the smallest pieces, then combine them in ways that almost immediately cease to be humanly comprehensible. Completely different selection mechanism for what appears, but it turns out Turing Machine states can do a lot "more" than you might think simply by looking at human-designed TMs. We humans have very stereotypical design methodologies and they have their advantages, but sometimes just letting algorithms rip can result in much better things than we could ever hope to design with the same resources.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#140

Post by yieldcrv »

turnsout wrote: post #48
yieldcrv wrote: post #18

This is great, my employer just gave me a M1 laptop with only 16gb ram and I had to downgrade my 7B parameter local LLM’s to 3 bit quantizing, they’ve been surprisingly okay! In my personal machine at 64gb ram, I usually use 8x7B at Q5 or 70B at Q4 Its Mistral all the way down! Imagining Q1.58 that’s doing well makes me happy

Quantized 7B LLMs should work fine on your machine, though maybe you’re talking about speed?

7B works fine
Post reply on HN