Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BitNet: Inference framework for 1-bit LLMs

github.com

Post reply on HN
Print view View original

21–30 of 179 posts

Re: BitNet: Inference framework for 1-bit LLMs

#21

Post by Dwedit »

drsopp wrote: post #8
nickcw wrote: post #7

> bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU and GPU (NPU support will coming next). One bit or one trit? I am confused!

"1-bit LLMs" is just marketing. The Shannon entropy of one letter with a 3 symbol alphabet (-1, 0, 1) is 1.58.

Log Base 2 of 3 = ~1.5849625, so that's the limit to how well you can pack three-state values into bits of data.

For something more practical, you can pack five three-state values within a byte because 3^5 = 243, which is smaller than 256. To unpack, you divide and modulo by 3 five separate times. This encodes data in bytes at 1.6 bits per symbol.

But the packing of 5 symbols into a byte was not done here. Instead, they packed 4 symbols into a byte to reduce computational complexity (no unpacking needed)

Re: BitNet: Inference framework for 1-bit LLMs

#22

Post by uniq7 »

giancarlostoro wrote: post #14

One of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think.…

Since Google Search already includes an AI summary, your minimally viable "LLM" can be just an HTTP GET call

Re: BitNet: Inference framework for 1-bit LLMs

#23

Post by orbital-decay »

cubefox wrote: post #17
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

LLM account

Funny enough I now involuntarily take RTFA as a slight slop signal, because all these accounts dutifully read the article before commenting, unlike most HNers who often respond to headlines.

Re: BitNet: Inference framework for 1-bit LLMs

#24

Post by intrasight »

giancarlostoro wrote: post #14

One of the things I often wonder is "what will be the minimally viable LLM" that can work from just enough information that if it googles the rest it can provide reasonable answers? I'm surprised something like Encyclopedia Britanica hasn't yet (afaik) tried to capitalize on AI by selling their data to LLMs and validating outputs for LLM companies, it would make a night and day difference in some areas I would think.…

It's not so much a "minimally viable LLM" but rather an LLM that knows natural language well but knows nothing else. Like me - as an engineer who knows how to troubleshoot in general but doesn't know about a specific device like my furnace (recent example).

And I don't think that LLM could just Google or check Wikipedia.

But I do agree that this architecture makes a lot of sense. I assume it will become the norm to use such edge LLMs.

Re: BitNet: Inference framework for 1-bit LLMs

#25

Post by WithinReason »

LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

> a fundamentally different compute profile on commodity CPU

In what way? On modern processors, a Fused Multiply-Add (FMA) instruction generally has the exact same execution throughput as a basic addition instruction

Re: BitNet: Inference framework for 1-bit LLMs

#26

Post by wongarsu »

rustyhancock wrote: post #12
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

Yes. I had to read it over twice, it does strike me as odd that there wasn't a base model to work with. But it seems the biggest model available is 10B? Somewhat unusual and does make me wonder just how challenging it will be to train any model in the 100B order of magnitude.

Approximately as challenging as training a regular 100B model from scratch. Maybe a bit more challenging because there's less experience with it

The key insight of the BitNet paper was that using their custom BitLinear layer instead of normal Linear layers (as well as some more training and architecture changes) lead to much, much better results than quantizing an existing model down to 1.58 bits. So you end up making a full training run in bf16 precision using the specially adapted model architecture

Re: BitNet: Inference framework for 1-bit LLMs

#27

Post by regularfry »

wongarsu wrote: post #15
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

I've also always though that it's an interesting opportunity for custom hardware. Two bit addition is incredibly cheap in hardware, especially compared to anything involving floating point. You could make huge vector instructions on the cheap, then connect it to the fastest memory you can buy, and you have a capable inference chip. You'd still need full GPUs for training, but for inference the hardware would be order…

You only need GPUs if you assume the training is gradient descent. GAs or anything else that can handle nonlinearities would be fine, and possibly fast enough to be interesting.

Re: BitNet: Inference framework for 1-bit LLMs

#28

Post by gregman1 »

embedding-shape wrote: post #16
LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

> Framework is ready. Now we need someone to actually train the model. If Microslop aren't gonna train the model themselves to prove their own thesis, why would others? They've had 2 years (I think?) to prove BitNet in at least some way, are you really saying they haven't tried so far? Personally that makes it slightly worrisome to just take what they say at face value, why wouldn't they train and publish a model the…

Cannot agree more!

Re: BitNet: Inference framework for 1-bit LLMs

#29

Post by yorwba »

orbital-decay wrote: post #23
cubefox wrote: post #17

Earlier quoted context omitted.

LLM account

Funny enough I now involuntarily take RTFA as a slight slop signal, because all these accounts dutifully read the article before commenting, unlike most HNers who often respond to headlines.

Not all of them do: https://news.ycombinator.com/item?id=47335156 There are evidently lots of people experimenting with different botting setups. Some do better at blending in than others.

Re: BitNet: Inference framework for 1-bit LLMs

#30

Post by august11 »

LuxBennu wrote: post #6

The title is misleading — there's no trained 100B model, just an inference framework that claims to handle one. But the engineering is worth paying attention to. I run quantized 70B models locally (M2 Max 96GB, llama.cpp + LiteLLM), and memory bandwidth is always the bottleneck. The 1.58-bit approach is interesting because ternary weights turn matmuls into additions — a fundamentally different compute profile on comm…

In their demo they're running 3B model.
Post reply on HN