Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

121–130 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#121

Post by dofm »

mandeepj wrote: post #101
segmondy wrote: post #87

Earlier quoted context omitted.

Apple is too desperate to be making demands. You confuse the Apple of yesterday and of today. Things change fast and things have changed.

> Apple is too desperate to be making demands They don't give a F about AI or any new AI model that was announced this morning. Wasn't there news a while ago about them buying Perplexity?

They didn't buy Perplexity and it indeed rather seems like Perplexity may have damaged the deal by blurting.

Whether things are different now I don't know, but Prism would potentially be a good acquisition.

This is not the first Bonsai model from Prism using this technology, and they've also applied this technology to an image model.

The model aspect isn't that significant (even in this news), because it is Qwen, under the hood. The encoding efficiency is.

Prism's technology might be valuable, and their team could well be.

And they very evidently do care a lot about efficient on-device AI; they just don't care about developing frontier cloud models.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#122

Post by dofm »

segmondy wrote: post #116
mandeepj wrote: post #101

Earlier quoted context omitted.

> Apple is too desperate to be making demands They don't give a F about AI or any new AI model that was announced this morning. Wasn't there news a while ago about them buying Perplexity?

They do, they are low key panicking.

I do not believe they are panicking, not least because I don't think they've finished adjusting Apple Silicon for the task; it will be very interesting to see what happens in the M6 and M7.

I think their strategy is broadly correct, actually; I think there's still a bit of scope for "sit and wait and do it right" here. But acquiring more edge AI tech and edge AI people would potentially be in their interest.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#123

Post by janalsncm »

SwellJoe wrote: post #59

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the an…

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#124

Post by dofm »

rogerkirkness wrote: post #67
CharlesW wrote: post #58

Earlier quoted context omitted.

Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.

Apple would punish him severely unless they cleared it in advance, it might be to their advantage for some reason (negotiating with Google for Gemma rights? idk).

I would be slightly surprised if anything Apple wanted to do with Gemma they couldn't do with Gemini, which they have the right to make various derivatives of.

They could more or less redistribute Gemma as-is in the developer program; they are unlikely to be troubled by any of the licence terms.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#125

Post by SwellJoe »

simonw wrote: post #8

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.

I got their previous model working in their custom fork of llama.cpp (https://github.com/PrismML-Eng/llama.cpp). I haven't tried this one yet, but will find some time to benchmark it sometime this week.

Though this says mainline llama.cpp has their patches for Metal and CPU backends, so maybe it's simply "use current llama.cpp" if you have a Mac or fast enough CPU/memor to use the CPU backend.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#126

Post by dofm »

verdverm wrote: post #107
dofm wrote: post #105

Earlier quoted context omitted.

The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better). The 12B QAT model is indeed sort of mindblowing.

I haven't dug into QAT deeply, better recovery is my understanding as well, and also that it is out of reach for most people because you have to train a model to back prop errors based on estimated error under quant. Hopefully more of the lab releases are trained under QAT so we can all benefit.

I think they did Gemma 3 QAT models and there are QAT versions of essentially all the Gemma 4 models (including DiffusionGemma).

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#127

Post by dofm »

simonw wrote: post #8

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.

Not sure if there's any way to run Prism's fork of llama.cpp inside LM Studio.

The fork runs fine for me. The model gets very notably stuck in a reasoning loop on one of my simple tests, though it might be that it has the same issues with setting reasoning effort high.

On my M1 Max I still think the MoE Qwen 3.6 and Gemma 4 models are the best options. And I am far from convinced that the 35B is actually worse; it gets stuck in reasoning loops much less often than 27B in my experience.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#128

Post by novaRom »

dofm wrote: post #105
verdverm wrote: post #70

Earlier quoted context omitted.

4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn. Good evaluation from 2024 https://arxiv.org/pdf/2402.18158 I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I…

The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better). The 12B QAT model is indeed sort of mindblowing.

Gemma 4 12B QAT is amazing - agents run very fast, and it's really very smart, at least in my agent's harness domain which is GNU software development - on par with frontiers like GPT Sol, DeepSeek, or Claude - Why to buy those expensive tokens if a local tiny model performs so well?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#129

Post by CuriouslyC »

janalsncm wrote: post #123
SwellJoe wrote: post #59

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the an…

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

The things it loses are all the things that google models are historically excellent at, so that's a reasonable performance. I think the take home here is that the 1 bit models are probably better, but it's not a slam dunk given advanced quantization techniques.
Post reply on HN