Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

171–180 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#171

Post by gwerbin »

novaRom wrote: post #128
dofm wrote: post #105

Earlier quoted context omitted.

The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better). The 12B QAT model is indeed sort of mindblowing.

Gemma 4 12B QAT is amazing - agents run very fast, and it's really very smart, at least in my agent's harness domain which is GNU software development - on par with frontiers like GPT Sol, DeepSeek, or Claude - Why to buy those expensive tokens if a local tiny model performs so well?

What's your harness setup? I haven't had this kind of on-par success with any local LLM yet.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#173

Post by pylotlight »

kamranjon wrote: post #80
RALaBarge wrote: post #68

Earlier quoted context omitted.

What have you been using?

DeepSeek V4 Flash with DwarfStar: https://github.com/antirez/ds4 The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.

But isn't it running basically 1 request at a time? This would make agentic coding difficult right? Compared to running as many sequential tests as you want via api?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#174

Post by SwellJoe »

ckluis wrote: post #147
SwellJoe wrote: post #59

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the an…

I'd like to see them do a 1-bit binary version of Gemma 4 12B ;)

I want 31B. The 12B 4-bit QAT is already small enough to run well enough on every device I use regularly, including phone and tablet, I don't need a 1-bit or ternary version of the 12B.

But, what I really want is for Google to release bigger Gemma 4 models, particularly a bigger MoE, like a ~70B or ~120B. Gemma 4 is the best all-rounder among the models I can self-host even though I've got a 128GB Strix Halo. A 4-bit QAT version of a 70B MoE would probably be the sweet spot.

A bigger Qwen 3.6 with a 4-bit QAT version would also be welcome, as the prior bigger versions aren't notably better than 3.6 27B, but I guess Qwen is done doing larger open weights models. They did release AgentWorld recently, a post-train of the 3.6 MoE, so they're still doing some open things.

I think I want to see more third-party testing of this ternary Qwen to know if crushing it to 1.56 bits kills it; there are tons of benchmarks of Qwen 3.6 27B, so it's an ideal candidate to figure out what the extreme compression does to it.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#175

Post by sosodev »

dofm wrote: post #157
janalsncm wrote: post #123

Earlier quoted context omitted.

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

FWIW my tests on my little puzzles suggest that it is not better than the Gemma 4 12B on SQL. It really does seem to get quite tangled up on stuff. PHP/Wordpress code seems OK (better than the Gemma) but it gets stuck in reasoning loops. Mind you, I am something of a cynic about the underlying 27B dense Qwen; I think the 35B MoE model is often better and it is just so, so much faster.

Qwen3.6-27B is the best model in that range that I’ve used for agentic coding by far. I think it’s kinda mid at everything else.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#176

Post by sosodev »

gwerbin wrote: post #171
novaRom wrote: post #128

Earlier quoted context omitted.

Gemma 4 12B QAT is amazing - agents run very fast, and it's really very smart, at least in my agent's harness domain which is GNU software development - on par with frontiers like GPT Sol, DeepSeek, or Claude - Why to buy those expensive tokens if a local tiny model performs so well?

What's your harness setup? I haven't had this kind of on-par success with any local LLM yet.

They’re exaggerating or have a very simple way of using these models. The Gemma 4 series, even at 31B, is nowhere near the frontier. They’re great models, but you will notice a huge difference for complex tasks.

The best local agentic coding experience I’ve had so far is Qwen3.6-27B with Pi.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#177

Post by anon373839 »

simonw wrote: post #8

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.

I can report that it's working in oMLX. I've been experimenting with the ternary one; it is quite an impressive model! I've been grilling it on some deep learning/computer vision stuff and it's aced everything so far. Responses are thorough, accurate, sophisticated. General knowledge outside of CS doesn't seem as robust, which I expected. Honestly, I don't think the examples in the blog post do it justice.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#178

Post by verdverm »

sosodev wrote: post #176
gwerbin wrote: post #171

Earlier quoted context omitted.

What's your harness setup? I haven't had this kind of on-par success with any local LLM yet.

They’re exaggerating or have a very simple way of using these models. The Gemma 4 series, even at 31B, is nowhere near the frontier. They’re great models, but you will notice a huge difference for complex tasks. The best local agentic coding experience I’ve had so far is Qwen3.6-27B with Pi.

same with opencode, next on my evals.list are the fine tunes from big model traces, and then my own if ftuning looks reasonable cost/time wise

I have two main tasks I want to see if I can improve, coding and doc understanding/summary

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#179

Post by yieldcrv »

this is really amazing! keep pushing guys, this will coexist in the memory footprint with vision models, and audio models, and other kinds of transformers so we still need memory to work with

you also might single handedly pop the hyperscaler investment and capital projects! that's the whole AI bubble essentially!

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#180

Post by kennywinker »

bigmadshoe wrote: post #144
recursivegirth wrote: post #133

Earlier quoted context omitted.

I think this is where leveraging classifier models will become important. The frontier LLM models do "everything", while we've known for a while that to truly scale this we will need to distill models into their individual functions. I don't see this as necessarily a bad thing and hope more is done in this space. Very promising.

Bitter lesson is knocking. Mixture of experts is essentially what you’re describing but free from unnecessary inductive biases.

There is value in splitting things. If all I ever do is local app automations, i don’t need model that knows how to code. If all I ever do is coding, i don’t need a model that translates english to slovakian.
Post reply on HN