[deleted]
BitNet b1.58 2B4T Technical Report
21–30 of 33 posts
Re: BitNet b1.58 2B4T Technical Report
#22I built it at home this morning and tried it, perhaps my expectations were high but I wasn't terribly impressed. I asked it for a list of ten types of data I might show on a home info display panel. It gave me three. I clarified that I wanted ten, it gave me six. Every request after that just returned the same six things. I know it's not chatGPT4 but I've tried other very small models that run on CPU only and had bet…
> I've tried other very small models that run on CPU only and had better results Maybe you can you share some comparative examples?
sure, here's my conversation with BitNet b1.58 2B4T
here's the same prompt given to smollm2:135m
The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet
Smollm2:360M returned better quality results, no repetition, but it did suggest things which didn't fit the brief exactly (public data given location only)
Edit:
I tried the same query on the live demo site and got much better results. Maybe something went wrong on my end?
Re: BitNet b1.58 2B4T Technical Report
#23Is there a reason why the 1.58 ones are always aimed at quite small ones? Think I’ve seen an 8B but that’s about it. Is there a technical reason for it or just research convenience ?
They aren’t, there is a 1.58 version of deepseek that’s like 200gb instead of 700
Re: BitNet b1.58 2B4T Technical Report
#24Earlier quoted context omitted.
> I've tried other very small models that run on CPU only and had better results Maybe you can you share some comparative examples?
sure, here's my conversation with BitNet b1.58 2B4T https://pastebin.com/ZZ1tADvp here's the same prompt given to smollm2:135m https://pastebin.com/SZCL5WkC The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet Smollm2:36…
You were using bitnet.cpp?
Re: BitNet b1.58 2B4T Technical Report
#25Earlier quoted context omitted.
sure, here's my conversation with BitNet b1.58 2B4T https://pastebin.com/ZZ1tADvp here's the same prompt given to smollm2:135m https://pastebin.com/SZCL5WkC The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet Smollm2:36…
You were using bitnet.cpp?
Yes
Re: BitNet b1.58 2B4T Technical Report
#26I ask about the last French election and the #1 sentence is:
>Marine Le Pen, a prominent figure in France, won the 2017 presidential election despite not championing neoliberalism. Several factors contributed to her success: (…)
What data did they train their model on?
Re: BitNet b1.58 2B4T Technical Report
#27They give some description of how their weights are stored: they pack 4 weights into an int8, indicating that their storage format isn't optimal (2 bits per weight instead of the optimal ~1.58 bits). But I don't know enough about LLM internals to know how material this is. Could anyone break down the steps further?
This model maps weights to ternary values {-1, 0, 1} (aka trits). One trit holds log(3)/log(2) ≈ 1.58 bits of information. To represent a single trit by itself would require 2 bits, but it is possible to pack 5 trits into 8 bits. This article explains it well: https://compilade.net/blog/ternary-packing
By using 4 ternary weights per 8 bits, the model is not quite as space-efficient as it could be in terms of information density. (4*1.58)/8 = 0.79 vs (5*1.58)/8 = 0.988 There is currently no hardware acceleration for doing operations on 5 trits packed into 8 bits, so the weights have to be packed and unpacked in software. Packing 5 weights into 8 bits requires slower, more complex packing/unpacking algorithms.
Re: BitNet b1.58 2B4T Technical Report
#28Not to be confused with BITNET
Re: BitNet b1.58 2B4T Technical Report
#29Is there a reason why the 1.58 ones are always aimed at quite small ones? Think I’ve seen an 8B but that’s about it. Is there a technical reason for it or just research convenience ?
They aren’t, there is a 1.58 version of deepseek that’s like 200gb instead of 700
That's not a real BitNet, it's just a post-training quantisation, and its performance suffers compared to if it was trained from scratch at 1.58 bits.
Re: BitNet b1.58 2B4T Technical Report
#30I guess B1FF@BITNET posts are gonna come from an LLM now.
Context: https://web.archive.org/web/20030830105202/http://www.catb.o...