Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

211–220 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#211

Post by larodi »

sosodev wrote: post #175
dofm wrote: post #157

Earlier quoted context omitted.

FWIW my tests on my little puzzles suggest that it is not better than the Gemma 4 12B on SQL. It really does seem to get quite tangled up on stuff. PHP/Wordpress code seems OK (better than the Gemma) but it gets stuck in reasoning loops. Mind you, I am something of a cynic about the underlying 27B dense Qwen; I think the 35B MoE model is often better and it is just so, so much faster.

Qwen3.6-27B is the best model in that range that I’ve used for agentic coding by far. I think it’s kinda mid at everything else.

Surely not that good at vision. TBH none of these 14-27b models come close to even the cheapest Gemma 2.5 flash.

If these buddies are similarly bad on text, then they definitely don’t get anywhere close to big boys, no matter what the synthetic stats claim upon release.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#213

Post by davedx »

I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neural networks.

I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a better choice?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#214

Post by snthpy »

edflsafoiewq wrote: post #54
PcChip wrote: post #26

Earlier quoted context omitted.

this is a really dumb question, but how is -1 represented? is it a float? if so, how many bits is the float? I've never heard of a bit ever having more than two possible values

It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128. Their fork corrects th…

Thanks. This relates to some questions I've got. I was playing around with the previous generation smaller models and found that i wasn't getting any speedups from the T1 and T2 binary/ternary models compared to standard Q4 quants of straight qwen3.6 models. I was wondering whether unpacking of the ternary encoding was impacting inference speed?

If that's the case then why not just train at Q2? I guess the counterargument is that then you lose the nice properties of things like the FairyFuse kernels. I wish there were some good discussions of these trade off.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#215

Post by Zababa »

kennywinker wrote: post #180
bigmadshoe wrote: post #144

Earlier quoted context omitted.

Bitter lesson is knocking. Mixture of experts is essentially what you’re describing but free from unnecessary inductive biases.

There is value in splitting things. If all I ever do is local app automations, i don’t need model that knows how to code. If all I ever do is coding, i don’t need a model that translates english to slovakian.

There is value in splitting things but there is also a cost. You have to train the specialized model, for that you have to know your use case, you have to hope the use case is going to be stable over time, you then have to see if you can remove english -> slovakian or coding from a model without affecting the useful parts.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#216

Post by int_19h »

kaycey2022 wrote: post #182
hypercube33 wrote: post #143

Earlier quoted context omitted.

More to the argument that we need a model of models - one general one that calls specialists in to do what they are good at and handles that like a foreman for you.

Is that different from mixture of experts?

Yes. A mixture of experts is a single model that activates different routes though the same weights, with the route possibly changing literally on every token. It's not experts as in a bunch of standalone models that are good at specific high-level tasks.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#217

Post by mehmetkose »

motbus3 wrote: post #46

I need help understanding this. I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other…

1-bit weights are not pointers so cpus can process them, storing them takes less space etc. tons of gains

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#218

Post by jboss10 »

davedx wrote: post #213

I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neural networks. I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a bette…

Most of the time, the speed of these models are constrained by memory bandwidth. GPUs normally have much more memory bandwidth.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#219

Post by snthpy »

What is the best way to deploy these on CPUs, arm64 ones in particular?

I'm interested in the CPU inference application of these models with things like the FairyFuse kernels.

I've tried trillim previously but was disappointed that i got higher tok/s just with similar sized models through ollama using just Q4_K_M quants.

I see there is bitnet.cpp and litespark-inference. What else should i look at?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#220

Post by SwellJoe »

moffkalast wrote: post #201
janalsncm wrote: post #123

Earlier quoted context omitted.

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

> slightly worse on knowledge and tool calling Worse than Gemma at tool calling? Gemma's already bottom tier at that (at least when there's Qwen to compare to), that would just be unable to do tool calling at all.

I think that depends on how you run it. Llama.cpp has several fixes for the somewhat unusual tool call semantics in Gemma 4. I don't think I have noticed any issues.
Post reply on HN