Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

221–230 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#221

Post by MrDrMcCoy »

snthpy wrote: post #219

What is the best way to deploy these on CPUs, arm64 ones in particular? I'm interested in the CPU inference application of these models with things like the FairyFuse kernels. I've tried trillim previously but was disappointed that i got higher tok/s just with similar sized models through ollama using just Q4_K_M quants. I see there is bitnet.cpp and litespark-inference. What else should i look at?

Llama.cpp should be rolling out support for this soon if they haven't already. Cactus is a but more targeted for efficient ARM execution, but I haven't been keeping up with what all they support. Would be worth a feature request if they aren't working on it yet.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#222

Post by thomasjb »

kennywinker wrote: post #183
thomasjb wrote: post #17

I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)

Toss the rtx into a cheapo optiplex or thinkcenter, and run it headless - the load on your machine is gonna make doing other stuff while it’s running painful. Plus that frees up the rest of your vram.

I aspire to someday move up to an AM5 system, but for now, the Dell T3610 has to do everything. Might get a second GPU for it though.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#223

Post by literalAardvark »

trollbridge wrote: post #119
segmondy wrote: post #116

Earlier quoted context omitted.

They do, they are low key panicking.

And if they aren't, they should be. They ship some of the cheapest, most capable, and easiest to buy/set up AI hardware out there, and then don't ship any AI software worth speaking of. Apple needs to fix this.

"Apple needs to get into gold prospecting, this funny business of selling the best shovels just can't work out"

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#224

Post by davedx »

jboss10 wrote: post #218
davedx wrote: post #213

I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neural networks. I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a bette…

Most of the time, the speed of these models are constrained by memory bandwidth. GPUs normally have much more memory bandwidth.

I'd expect the memory bandwidth to be the same for the CPU and GPU under a unified memory architecture like Apple silicon uses?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#225

Post by kmacdough »

davedx wrote: post #213

I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neural networks. I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a bette…

Its not about geometry, it's a parallel compute thing thing. CPUs typically don't have more than 10 or 20 cores. GPU have 100s to 1000s.

Matmul is very well parallelized. More, lower power cores will always pay off handsomely.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#226

Post by hovering_nox »

bigmadshoe wrote: post #144
recursivegirth wrote: post #133

Earlier quoted context omitted.

I think this is where leveraging classifier models will become important. The frontier LLM models do "everything", while we've known for a while that to truly scale this we will need to distill models into their individual functions. I don't see this as necessarily a bad thing and hope more is done in this space. Very promising.

Bitter lesson is knocking. Mixture of experts is essentially what you’re describing but free from unnecessary inductive biases.

Mixture of Experts is absolutely not what they're describing. MoE has to be one of the most misleadingly named things ever. It's completely confusing as to what it actually is.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#227

Post by xienze »

moffkalast wrote: post #201
janalsncm wrote: post #123

Earlier quoted context omitted.

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

> slightly worse on knowledge and tool calling Worse than Gemma at tool calling? Gemma's already bottom tier at that (at least when there's Qwen to compare to), that would just be unable to do tool calling at all.

I think it's extremely quantization and engine specific. I run Gemma4-31B at FP8 on vLLM and it's fantastic, no issues anymore[*].

* I will say that early on there were a LOT of issues with the chat template, across all engines. I dunno who decided using crappy Jinja templates was a good idea, but clearly it has its limitations. In the latest version of vLLM (0.25) they've ditched the Jinja templates for an in-engine parser and I've seen no issues.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#228

Post by ralusek »

hovering_nox wrote: post #226
bigmadshoe wrote: post #144

Earlier quoted context omitted.

Bitter lesson is knocking. Mixture of experts is essentially what you’re describing but free from unnecessary inductive biases.

Mixture of Experts is absolutely not what they're describing. MoE has to be one of the most misleadingly named things ever. It's completely confusing as to what it actually is.

MoE is literally exactly what they're describing. The classifier being described is baked into the model and is the thing that makes it MoE.

Imagine I had [a model that was good at math], [model that was good at code], [model that was good at writing], [model that was good at general knowledge]. If I then had [a model that was good at determining whether the user query would be best served by one of those models and sent it to it, leaving the rest of the models inactive], that is the platonic version of what MoE is. In practice, it works a bit differently. It instead basically restricts the number of pathways that can be utilized in solving problems during training, which allows for "expert neuron groupings" to form and "classifier layers" to form earlier on in the structure, but the effect is the same (better, even, since it allows some overlap between structures of experts). It also allows "routing to an expert" to happen token-by-token rather than at the prompt level.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#230

Post by xienze »

sosodev wrote: post #176
gwerbin wrote: post #171

Earlier quoted context omitted.

What's your harness setup? I haven't had this kind of on-par success with any local LLM yet.

They’re exaggerating or have a very simple way of using these models. The Gemma 4 series, even at 31B, is nowhere near the frontier. They’re great models, but you will notice a huge difference for complex tasks. The best local agentic coding experience I’ve had so far is Qwen3.6-27B with Pi.

> They’re great models, but you will notice a huge difference for complex tasks.

Yes and no. I think where frontier models really blow small models away is in how thorough they are in order to infer your intentions and how best to accomplish them. So you can tell Claude "change this code to make it do X", whereas a Qwen3.6-27B or Gemma4-31B can do the same, but you have to be a lot more thorough, i.e., "change this code to make it do X, but first, let me explain the concept of X as I see it and some notes about things to avoid or pay attention to while you're doing it." So for best success with small models you really need a big toolbelt of skills and MCPs.

Post reply on HN