BitNet b1.58 2B4T Technical Report • Hacker News Board

Re: BitNet b1.58 2B4T Technical Report

#31

Post by Me1000 » Thu, Apr 17, 2025, 6:46 PM UTC

nopelynopington wrote: post #17

I built it at home this morning and tried it, perhaps my expectations were high but I wasn't terribly impressed. I asked it for a list of ten types of data I might show on a home info display panel. It gave me three. I clarified that I wanted ten, it gave me six. Every request after that just returned the same six things. I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had bet…

This is a technology demo, not a model you'd want to use. Because Bitnet models are only average 1.58 bits per weight you'd expect to need the model to be much larger than your fp8/fp16 counterparts in terms of parameter count. Plus this is only a 2 billion parameter model in the first place, even fp16 2B parameter models generally perform pretty poorly.

Re: BitNet b1.58 2B4T Technical Report

#32

Post by nopelynopington » Thu, Apr 17, 2025, 7:11 PM UTC

Me1000 wrote: post #31

nopelynopington wrote: post #17

I built it at home this morning and tried it, perhaps my expectations were high but I wasn't terribly impressed. I asked it for a list of ten types of data I might show on a home info display panel. It gave me three. I clarified that I wanted ten, it gave me six. Every request after that just returned the same six things. I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had bet…

This is a technology demo, not a model you'd want to use. Because Bitnet models are only average 1.58 bits per weight you'd expect to need the model to be much larger than your fp8/fp16 counterparts in terms of parameter count. Plus this is only a 2 billion parameter model in the first place, even fp16 2B parameter models generally perform pretty poorly.

Ok that's fair. I still think something was up with my build though, the online demo worked far better than my local build

Re: BitNet b1.58 2B4T Technical Report

#33

Post by akoboldfrying » Thu, Apr 17, 2025, 11:27 PM UTC

Fubwubs wrote: post #27

akoboldfrying wrote: post #6

They give some description of how their weights are stored: they pack 4 weights into an int8, indicating that their storage format isn't optimal (2 bits per weight instead of the optimal ~1.58 bits). But I don't know enough about LLM internals to know how material this is. Could anyone break down the steps further?

This model maps weights to ternary values {-1, 0, 1} (aka trits). One trit holds log(3)/log(2) ≈ 1.58 bits of information. To represent a single trit by itself would require 2 bits, but it is possible to pack 5 trits into 8 bits. This article explains it well: https://compilade.net/blog/ternary-packing By using 4 ternary weights per 8 bits, the model is not quite as space-efficient as it could be in terms of informat…

That link gives a great description of how to pack trits more efficiently, thanks. Encoding in "base 3" was obvious to me, but I didn't realise that 5 trits fit quite tightly into a byte, or that it's possible to "space the values apart" so that they can be extracted using just multiplications and bitwise ops (no division or remainder).