Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

51–60 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#51

Post by h14h »

liuliu wrote: post #6
liuliu wrote: post #3

The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.

You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.

I'm curious what kind of results one could get from combining the clever quantization PrismML is doing here with something like LiquidAI's antidoom:

https://github.com/Liquid4All/antidoom

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#52

Post by 0xbadcafebee »

27B is way more than you need for a phone. Doesn't matter how much you try to compress it, it's the wrong application of the wrong tool. There are already useful tiny models that fit on phones and do basic things really well. Dumb down a big model too much and it becomes worse than a small fine-tuned model.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#54

Post by edflsafoiewq »

PcChip wrote: post #26
NitpickLawyer wrote: post #9

Earlier quoted context omitted.

There's two variants of this (or, as the joke goes, for very big values of bit): Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. 1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

this is a really dumb question, but how is -1 represented? is it a float? if so, how many bits is the float? I've never heard of a bit ever having more than two possible values

It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128.

Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.

It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#55

Post by liuliu »

gunalx wrote: post #34
liuliu wrote: post #13

Earlier quoted context omitted.

Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).

3.5 9B can do thinking. Its just disabled by default in its gguf chat template.

It is disabled because it doesn't work :) Try it and see the doom loop it gets itself in.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#56

Post by PrismML »

motbus3 wrote: post #47
bansaltushar wrote: post #29

Earlier quoted context omitted.

Depending on which model you're running, you might need to use the custom forks. Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....

I spent quite sometime trying to install their tools and nothing really worked. I used these repos you shared but the dependencies all fail on mac

If you can share details on where it is failing, we'd love to help fix.

You can also join Discord to communicate with us directly http://discord.gg/prismml

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#57

Post by wy35 »

wmf wrote: post #42
wy35 wrote: post #39

Entire blog post seems to be AI-generated :/

Do you think people who work on AI for a living are not going to use it?

Of course not, personally almost all of my code these days is generated.

The LLM style of writing is just very distracting to read. “It unlocks X”, “Y changes the equation”, and why is there always something shifting? Makes my eyes glaze over in an otherwise interesting post.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#58

Post by CharlesW »

kristianp wrote: post #21

Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...

Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#59

Post by SwellJoe »

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the answer is "very little" at four bits.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#60

Post by arjie »

wy35 wrote: post #57
wmf wrote: post #42

Earlier quoted context omitted.

Do you think people who work on AI for a living are not going to use it?

Of course not, personally almost all of my code these days is generated. The LLM style of writing is just very distracting to read. “It unlocks X”, “Y changes the equation”, and why is there always something shifting? Makes my eyes glaze over in an otherwise interesting post.

The text is mostly content-free. Headline + charts are enough for most HN stories.
Post reply on HN