Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

441–450 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#441

Post by thesz »

igleria wrote: post #379
Solvency wrote: post #305

Earlier quoted context omitted.

Why isn't John Carmack working for OpenAI? Hell, why did he waste years at Meta to work on a VR headset and NOT AI? He even announced he wants to focus on AGI but he missed out on literally all the action.

he has his own AGI startup now https://dallasinnovates.com/john-carmacks-keen-technologies-... TBH I think they won't get anywhere. Doing good game engine work... why that would translate to AGI?

Let me introduce you to the wonderful game that is The Talos Principle: https://en.wikipedia.org/wiki/The_Talos_Principle

It discusses whether it is possible to evolve AGi using... computer game engine! And that is John's bread and butter.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#442

Post by danielmarkbruce »

edflsafoiewq wrote: post #397
bick_nyers wrote: post #392

Earlier quoted context omitted.

Extraordinary claims require extraordinary evidence. That's not to say that a 70B model is necessary, but surely something larger than 3B is doable, especially given that the results of the paper directly imply a significant reduction in memory requirements for training such a model.

> results of the paper directly imply a significant reduction in memory requirements for training such a model Isn't memory use in training higher, since they maintain high precision latent weights in addition to the binarized weights used in the forward pass?

Yes. The optimizer is keeping a higher precision copy. It's likely slower and requires more memory than an equivalent full precision model when it comes to training. I'd also imagine it requires a multiple of epochs to get one epoch equivalent because the forward pass will need several goes to get the right choice between three states, rather than just moving a little bit in the right direction.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#443

Post by danielmarkbruce »

thesz wrote: post #283
mise_en_place wrote: post #191

Earlier quoted context omitted.

Intuitively I've always been a bit skeptical of quantization. Wouldn't there be a tiny loss in precision by doing this type of quantization? I could imagine the error function increasing by utilizing these types of techniques.

John Carmack pointed out (and I learned it here at HN) that what training really needs is the *sign" of each individual gradient parameter. I.e., you can quantize gradient to -1, 0 and 1 and still have neural network learn much of the dataset.

They seem to be doing training with higher precision. The optimizer is keeping a copy.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#445

Post by rep_lodsb »

rep_lodsb wrote: post #440
phkahler wrote: post #412

Earlier quoted context omitted.

Fleshing out my thought above. If we want to multiply A*B = C and all operands are stored in 2 separate bits Ap and An (Ap = 1 if A = +1 while An = 1 if A = -1). We can do a product with: Cp = (Ap & Bp) | (An & Bn) Cn = (An & Bp) | (Ap & Bn) So 64 products in 6 instructions, or 256 in 6 instructions with AVX2, or 512 in six instructions using AVX512. If you can execute 2 instructions at a time on different words, thi…

Another way would be to use one register for "zero" vs. "non-zero", and another for negative (basically 2 bit sign-magnitude representation). C_sgn = A_sgn ^ B_sgn C_mag = A_mag & B_mag The result can then be converted into bitmasks for positive and negative: C_plus = C_mag & ~C_sgn C_minus = C_mag & C_sgn This solution should be more efficient if there is an "AND NOT" instruction, or when multiplying more than two f…

Thinking a bit more about this, you could eliminate the conversion and do

    sum = popcount(mag) - 2*popcount(mag & sgn)

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#446

Post by magnustesshu »

gemeral wrote: post #370
creshal wrote: post #118

Earlier quoted context omitted.

> * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. Easier said than done, of course, and very impressive that they pulled it off. > In matr…

> and blowing up parameter count to make up for it based on (an admittedly rapid and indulgent reading of the paper), it seems like they're not increasing the parameter size. Do you mind pointing out where the blowup is occurring?

They're saying that likely, models of comparable size will perform worse (the paper claims as good)

But since they are (optimized up to 8 or 10x if packing terns beyond 2 bits, in practice it seems 3-5x considering larger other structures needed in memory) more memory efficient, the largest models can be that much larger.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#447

Post by kirill5pol »

ordu wrote: post #417
eru wrote: post #340

Earlier quoted context omitted.

> So the proposition is: formal symbolisms can deal only with those problems that where already solved in imprecise human's languages. > To invent calculus and orbital mechanics you need first to talk for a several centuries (or thousands of years?) about what is position and velocity, you need to talk your way upto acceleration, and then you need to find a way to measure them and to define in a strict geometric term…

> all kinds of problems that you would never in a thousand years be able to handle with mere talk Not "all kinds of problems" but very specific kinds of problems which is possible to formalize into a math language. How would you go about inventing thermodynamics if you didn't know words "temperature" and "pressure"? You'd need to start for your senses that can tell you "this is a hot surface", or "this is a cold one"…

One possible way of looking at this is that human language is the way most people deal with abstraction, and abstract concepts. And there does seem to be some evidence that some of these abstractions in language may be universal to humans (I don’t fully buy all of the universal grammar stuff but still)

I think you could conceive of abstraction from other forms, maybe something like platonic forms as a base instead of language (again probably not in humans, but in others)

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#448

Post by nutanc »

licnep wrote: post #383
nutanc wrote: post #354

Earlier quoted context omitted.

Unfortunately the source code is currently not open sourced. Some more details at ( https://www.researchgate.net/publication/370980395_A_NEURAL_... ), the source code is built on top of this. The approach is used to solve other problems and papers have been published under https://www.researchgate.net/profile/K-Eswaran We are currently trying a build a full fledged LLM using just this approach(no LLM training etc) an…

Am I missing something or is this just a linear transformation? It says here ( https://www.researchgate.net/publication/370980395_A_NEURAL_... ) that each layer can be represented as a matrix multiplication (equation 3): Ax = s So concatenating multiple layers could just be reduced to a single matrix multiplication? If there is no non-linearity I don't see how this could replace neural networks, or am I missing somet…

The attempt is not to replace a particular neural network which has already been trained by using Sigmoid or Rel functions. If one does this then one would necessarily have to use non-linear maps. The whole point is that such a non-linear technique is not necessary for classifications. It is not necessary to confine clusters by hyperplanes for solving a classification problem. Our focus is on individual points.

We believe the brain does not do nonlinear maps!

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#449

Post by az226 »

rapatel0 wrote: post #435
az226 wrote: post #389

Earlier quoted context omitted.

What about text to speech models? Do you think ternary will work?

Just to be clear, it's all theoretically possible. There are already versions of BNN versions of YoLo and other CNNs. No reason why transformers wouldn't work for that or audio. It just might be harder to get them to train well enough. Speech to text, however, is super interesting. You just gave me an idea! I'm gonna go run some experiments :D

Please report back! :-)

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#450

Post by marty1885 »

ein0p wrote: post #268
ricardobeat wrote: post #243

Earlier quoted context omitted.

As someone else pointed out here, you can store 5 ternary values in 1 byte, 3^5 == 243.

That’s still not 1 bit, and that would basically destroy whatever perf advantage you might hope to get if you want to keep the model in memory in that format rather than unpack it on load.

Not fully, 8 bits has 256 values. It's easy to keep a look up table in the L1 cache of any CPU and constant cache of any GPU. For ASICs and FPGAs, it's a simple 256-value LUT. It's not ideal, yes, but not a deal breaker. Epically considering LLMs are memory bound. GGML dequantizes weights on-the-fly and still gets near linear scaling on GPUs.
Post reply on HN