Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Era of 1-bit LLMs: ternary parameters for cost-effective computing

arxiv.org

Post reply on HN
Print view View original

451–460 of 462 posts

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#451

Post by eru »

baq wrote: post #376
eru wrote: post #332

Earlier quoted context omitted.

Analog computing for neural networks is always very tempting. > We use binary states in normal computing to reduce entropy. In AI this is less of a concern, so why not use more of the available voltage range? Transistors that are fully closed or fully open use basically no energy: they either have approximately zero current or approximately zero resistance. Transistors that are partially open dissipate a lot of energ…

As someone who knows almost nothing about electronics I assume you’d want a transistor which can open in two ways: with positive and negative voltage. I’ve seen TNAND built out of normal transistors, not sure if such exotic ones would help even if they were physically possible.

That's for building ternary gates. They are still discrete, so it might be possible to do something here.

I was talking about analogue computing.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#452

Post by eru »

ordu wrote: post #417
eru wrote: post #340

Earlier quoted context omitted.

> So the proposition is: formal symbolisms can deal only with those problems that where already solved in imprecise human's languages. > To invent calculus and orbital mechanics you need first to talk for a several centuries (or thousands of years?) about what is position and velocity, you need to talk your way upto acceleration, and then you need to find a way to measure them and to define in a strict geometric term…

> all kinds of problems that you would never in a thousand years be able to handle with mere talk Not "all kinds of problems" but very specific kinds of problems which is possible to formalize into a math language. How would you go about inventing thermodynamics if you didn't know words "temperature" and "pressure"? You'd need to start for your senses that can tell you "this is a hot surface", or "this is a cold one"…

The 'mere talk' doesn't have to finish. You can have pretty nebulous ideas, and still start making progress with the formalism. The formalism can even help you 'finish' your thoughts.

In fact that kind of 'finishing' is very important, because otherwise you can waste a lot of time talking without noticing that you are not going anywhere. See eg philosophy or theology or pre-scientific-revolution science (ie natural philosohpy and natural history).

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#453

Post by eru »

the8472 wrote: post #393
eru wrote: post #337

Earlier quoted context omitted.

Yes, but the whole point of the link submitted to HN here is that in some applications, like machine learning, precision doesn't matter too much. (However, analog computing is still a bad fit for machine learning, because it requires a lot more power.)

Exact copies aren't just about precision but also about reproducibility.

You can keep your weights in a discrete format for storage, but do inference and training in analog.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#455

Post by HanClinto »

bilsbie wrote: post #238

This really just sounds absurd. How can ternary possibly encode enough information? Anyone willing to explain it like I’m a Django developer who watched half a karpathy video?

On its own, each trit doesn't encode much information at all. But it's not about information at the individual level -- it's more about the shape of the network.

I appreciated this comment [0] from earlier in the thread by paul_mk1:

> My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.

For myself, I've done a lot of work with image hashing (such as pHash and dHash) -- and in those, you throw away a LOT of information, but simply by keeping the value of each region and tracking whether or not it's above or below the average (essentially, the sign), then it's astounding how robust those algorithms are. Because you don't look at the individual pixels of an image, but it's very good at capturing the impression of the overall _shape_ of the image.

It's less about each individual datum, and more about the shape of the network.

If you're not familiar with Lottery Ticket Hypothesis, that would be worth reading up on.

[0]: https://news.ycombinator.com/item?id=39544500

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#456

Post by khimaros »

SushiHippie wrote: post #180
nutanc wrote: post #122

Earlier quoted context omitted.

We have been experimenting with the paper( https://www.researchgate.net/publication/372834606_ON_NON-IT... ). There is a mathematical proof that binary representation is enough to capture the latent space. And in fact we don't even need to do "training" to get that representation. The practical application we tried out for this algorithm was to create an alternate space for mpnet embeddings of Wikipedia paragraphs. U…

Wow, this works better than I would've thought. > Who moderates Hacker News? First result: > Hacker News > At the end of March 2014, Graham stepped away from his leadership role at Y Combinator, leaving Hacker News administration in the hands of other staff members. The site is currently moderated by Daniel Gackle who posts under the username "dang".

how did you test this?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#457

Post by SushiHippie »

khimaros wrote: post #456
SushiHippie wrote: post #180

Earlier quoted context omitted.

Wow, this works better than I would've thought. > Who moderates Hacker News? First result: > Hacker News > At the end of March 2014, Graham stepped away from his leadership role at Y Combinator, leaving Hacker News administration in the hands of other staff members. The site is currently moderated by Daniel Gackle who posts under the username "dang".

how did you test this?

First link in the substack article

https://speech-kws.ozonetel.com/wiki

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#458

Post by the8472 »

eru wrote: post #453
the8472 wrote: post #393

Earlier quoted context omitted.

Exact copies aren't just about precision but also about reproducibility.

You can keep your weights in a discrete format for storage, but do inference and training in analog.

That only prevents analog copy degradation. It doesn't give you reproducibility. Reproducibility means running the same process twice with the same inputs and getting the same outputs. E.g. to later prove that something came from an LLM and not a human you could store the random seed and the input and then reproduce the output. But that only works if the network is digital.

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#459

Post by stolsvik »

eru wrote: post #325
sandyarmstrong wrote: post #202

Earlier quoted context omitted.

LLMs and vector embeddings are always lossy compression, yes?

Almost always. Though you can use them in a lossless compression system, too, with a few tricks.

.. but you don't want to tell us?

Re: The Era of 1-bit LLMs: ternary parameters for cost-effective computing

#460

Post by eru »

stolsvik wrote: post #459
eru wrote: post #325

Earlier quoted context omitted.

Almost always. Though you can use them in a lossless compression system, too, with a few tricks.

.. but you don't want to tell us?

Two possible implementation:

(1) Take your data as a stream. Use your machine learning gadget to give you the (predicted) probability for each of the possible next tokens. Then use those probability in arithmetic coding to specify which token actually came next.

(2) Take your data D. Apply lossy compression to it. Store the result L := lossy(D). Also compute the residue R := D - uncompress(L). If your lossy compression is good, R will be mostly zeroes (and only a few actually differences), so it will compress well with a lossless compression algorithm.

Approach (1) is a more sophisticated version of (2). None of this is anything I came up with, those approaches are well known.

See eg https://arxiv.org/abs/2306.04050 and https://en.wikipedia.org/wiki/Audio_Lossless_Coding or https://ietresearch.onlinelibrary.wiley.com/doi/full/10.1049... (Probably not the best links, but something I could find quickly.)

Post reply on HN