Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

331–340 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#331

Post by dwightboyyy »

loudmax wrote: post #63
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

The Veritasium Youtube channel did a video about this about a year ago: https://www.youtube.com/watch?v=GVsUOuSjvcg They visit Texas company Mythic AI to discuss how they use flash memory for machine learning. There's a California company named Syntiant doing something similar.

I was thinking of this exact video, crazy to think that the principle is gaining momentum

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#332

Post by eru »

osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

Analog computing for neural networks is always very tempting.

> We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range?

Transistors that are fully closed or fully open use basically no energy: they either have approximately zero current or approximately zero resistance.

Transistors that are partially open dissipate a lot of energy; because they have some current flowing at some resistance. They get hot.

In addition, modern transistors are so small and so fast that the number of electrons (or holes..) flowing through them in clock cycle is perhaps in the range of a few dozen to a hundred. So that gives you at most 7 bits (~log_2(128)) of precision to work with in an analog setting. In practice, quite a bit less because there's a lot of thermal noise. Say perhaps 4 bits.

Going from 1 bit per transistor to 4 bits (of analog precision) is not worth the drastically higher energy consumption nor the deviation from the mainstream of semi-conductor technological advances.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#333

Post by GaggiX »

pclmulqdq wrote: post #311
GaggiX wrote: post #293

Earlier quoted context omitted.

According to the author, the 70B model is not fully trained.

"Is not fully trained" can also mean "we did not figure out how to reach an acceptable loss" or "training was unstable," both of which are common for ML systems.

It probably means that the model is not fully trained, because it is very expensive to train a 70B model, not even Mamba or RWKV have a model that comes close to that size, the leeriness is just kinda silly honestly.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#334

Post by eru »

gryn wrote: post #77
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

the reason why digital/numeric processing won is the power loss in the analog world. when design an analog circuit the next processing stage you add at the end has impact on the ones before it. this then require a higher skill from the engineers/consumers. if you want to avoid that you need to add op-amps with a gain of 1 at the boundary of each one, this also that care of the power loss at each stage. the other part…

You also have literal power losses, as in waste heat, to deal with.

See https://news.ycombinator.com/item?id=39545817

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#335

Post by eru »

int_19h wrote: post #218
trebligdivad wrote: post #170

Earlier quoted context omitted.

Trinary however is an interesting middle; people have built trinary hardware long ago; it feels like you could make natively trinary hardware for something like this; it might even be quite a win.

People haven't built reliable ternary electronics, though. Soviets tried with Setun, but they eventually had to resort to emulating each trit with two hardware bits (and wasting one state out of the possible four).

If you are are using two bits anyway, you might as well represent (-2, -1, 0, 1) instead of ternary?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#337

Post by eru »

the8472 wrote: post #127
osigurdson wrote: post #44

I have often mused that, in some ways, it seems like the transistor is really being wasted in AI applications. We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Basically, re-think the role of the transistor and re-design from the ground up - maybe NAND gates are not the ideal fundamental building block here?

Bits are copyable without data loss. Analog properties of individual transistors are less so.

Yes, but the whole point of the link submitted to HN here is that in some applications, like machine learning, precision doesn't matter too much.

(However, analog computing is still a bad fit for machine learning, because it requires a lot more power.)

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#338

Post by eru »

w-m wrote: post #119

I was reading Exposing Floating Point today (as Airfoil is on the HN front page and I was perusing the archive of the author). It's a blog explaining the inner workings of floating point representations. About zero values it says [0]: > Yes, the floating point standard specifies both +0.0 and −0.0. This concept is actually useful because it tells us from which “direction” the 0 was approached as a result of storing v…

You might also use a basis of negative-two and use two bits to represent {-2, -1, 0, 1}.

Negative bases are fun. See https://en.wikipedia.org/wiki/Negative_base

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#339

Post by eru »

fabiospampinato wrote: post #327
w-m wrote: post #119

I was reading Exposing Floating Point today (as Airfoil is on the HN front page and I was perusing the archive of the author). It's a blog explaining the inner workings of floating point representations. About zero values it says [0]: > Yes, the floating point standard specifies both +0.0 and −0.0. This concept is actually useful because it tells us from which “direction” the 0 was approached as a result of storing v…

I would guess that having 2 zeros is not that useful for NNs, but in general with 2 bits we could encode 4 states, so are there 4 possible states that would be useful to encode? Sure, but would this be better than encoding 3 states? That's the entire question imo. I would guess that 3 states are probably better, because negative/neutral/positive seems the minimal signal that we need these weights to provide.

You could use a negative-two base, and encode {-2, -1, 0, 1}. See https://en.wikipedia.org/wiki/Negative_base

Or you could use the regular positive-two base and encode {-2, -1, 0, 1} the normal way with two's complement.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#340

Post by eru »

ordu wrote: post #252
jart wrote: post #60

Earlier quoted context omitted.

LLMs will give normal people a firmer standing in technological society. That's a good thing. But will it change everything? Not a chance. Even if LLMs did change everything, that probably would not be a good thing. Dijkstra says Muslim algebra died when it returned to the rhetoric style, and the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters…

> the modern civilized world could only emerge —for better or for worse— when Western Europe could free itself from the fetters of medieval scholasticism I can propose an alternate view of things. Not that I'm going to argue that it is the only true statement in the world, but I think it is necessary for a thought to progress to have an alternative hypothesis. So the proposition is: formal symbolisms can deal only wi…

> So the proposition is: formal symbolisms can deal only with those problems that where already solved in imprecise human's languages.

> To invent calculus and orbital mechanics you need first to talk for a several centuries (or thousands of years?) about what is position and velocity, you need to talk your way upto acceleration, and then you need to find a way to measure them and to define in a strict geometric terms. Ah, and infinity, it was a very counter-intuitive idea, Xenon invented some of his paradoxes specifically to point at counter-intuitiveness. When Newton came all these talks and debates did the most of work for him.

For the sake of argument, let's grant your story about what you need to invent calculus.

But once you invented calculus, you can then use it to solve all kinds of problems that you would never in a thousand years be able to handle with mere talk.

Post reply on HN