Bonsai 27B: A 27B-Class model that runs on a phone
1–10 of 278 posts
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#2TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#3The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#4TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#5[dead]
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#6The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#7This must be some sort of unpublished app?
I can just see their image tool on the app store
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#8The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models
I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#9TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
There's two variants of this (or, as the joke goes, for very big values of bit):
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#10That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?