Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

61–70 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#61

Post by digdugdirk »

kamranjon wrote: post #53

After using a highly capable 2-bit quant as my daily driver for months now, I get pretty excited about releases like this. After a few days for the kinks to be worked out, I’ll be excited to try it.

What model? And what hardware do you run it on?

I find these style of models are great, but fail hard, and fail randomly. I'd be hesitant to use it for a daily driver, but I'm using dual 3060s, so it's not like I'm quantizing a frontier model here.

How do you find the overall experience? And do you have any special sauce or recommendations for going this route?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#62

Post by verdverm »

Preliminary analysis via lm-evaluation-harness + vllm

    model         | disk | wikitext | gsm8k (match/error)
    baseline      | 55G  | 8.00     | 0.50/0.09
    nvfp4-gptq    | 27G  | 8.25     | 0.47/0.9
    nvfp4a16-gptq | 27G  | 8.11     | 0.53/0.9
    bonsai-4bit   | 19G  | 16.75    | 0/0 (eval bug?)
Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. I'm also not sure what is up with the gsm8k, their benchmarks show something different, but they are using another eval tool. I'll have to add it to my setup. Also why I'm building a setup instead of taking model devs word for benchmarks. (https://github.com/modelscope/evalscope)

Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)

Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#63

Post by motbus3 »

PrismML wrote: post #56
motbus3 wrote: post #47

Earlier quoted context omitted.

I spent quite sometime trying to install their tools and nothing really worked. I used these repos you shared but the dependencies all fail on mac

If you can share details on where it is failing, we'd love to help fix. You can also join Discord to communicate with us directly http://discord.gg/prismml

I did not know you guys would be watching. For sure. Let me do that tomorrow when I turn it on again :) I am happy to see the message. Thanks!

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#65

Post by lioeters »

PcChip wrote: post #26
NitpickLawyer wrote: post #9

Earlier quoted context omitted.

There's two variants of this (or, as the joke goes, for very big values of bit): Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. 1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

this is a really dumb question, but how is -1 represented? is it a float? if so, how many bits is the float? I've never heard of a bit ever having more than two possible values

> never heard of a bit ever having more than two possible values

It's not represented by a "bit", binary digit with value of 0 or 1; but with a "trit", ternary digit with value of {−1, 0, +1}.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#66

Post by drob518 »

Arcuru wrote: post #43

Awesome! I've been waiting for them to start scaling ternary models for over a year[1]. Excited to try it out, typical Qwen 27B is too heavy for me to run on my local hardware at reasonable speeds. [1] https://jackson.dev/post/dont-sleep-on-bitnet/

Same here. I’m excited to have a model that might be usable on a 16 GB laptop.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#67

Post by rogerkirkness »

CharlesW wrote: post #58
kristianp wrote: post #21

Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...

Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.

Apple would punish him severely unless they cleared it in advance, it might be to their advantage for some reason (negotiating with Google for Gemma rights? idk).

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#70

Post by verdverm »

SwellJoe wrote: post #59

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the an…

4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn.

Good evaluation from 2024 https://arxiv.org/pdf/2402.18158

I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I have yet to do a real lit search.

Post reply on HN