Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

151–160 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#151

Post by foolfoolz »

codebje wrote: post #138
sandworm101 wrote: post #120

Earlier quoted context omitted.

And why would i want such mundane questions to be handled by an AI on my phone? That sort of thing doesnt need AI, let alone a local one. Basic google search was answering those questions long ago. My point: phone-sized AI is only useful if it can do things that only AI can do. Can it ingest a document scanned by the phones camera? Can it translate in real time? I dont see how or why i would ever ask it for recipe ad…

If it can give me the recipe without 14 pages of backstory about how Nonna used to make it, it'd be satisfying a real need.

every single recipe blog site builder has a “jump to recipe” button at the top

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#153

Post by stamps »

gnabgib wrote: post #150
purerandomness wrote: post #145

Earlier quoted context omitted.

Good spaghetti has 15% protein, 15g per 100g. Even the lowest quality one has 12g/100g.

Pasta should be made of whole durum wheat.. which is 12g/100g. What's this "good" you speak of? There's certainly worse (soft white/red flour with germ/bran removed), but where do we find the +3g? https://en.wikipedia.org/wiki/Pasta

I know felcetti has a few varieties that exceed that number. ~112g for 16g protein.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#154

Post by dofm »

trollbridge wrote: post #28
simonw wrote: post #8

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.

Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.

Have Prism-ML upstreamed their forked code?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#155

Post by Havoc »

smallerize wrote: post #134
Havoc wrote: post #132

Earlier quoted context omitted.

I've got both that app and a 17 Pro, but it only lists one of the older Bonsai models not the 27B for me

I only have an iPhone 14 Pro, but under "manage models" it's showing Bonsai 8B and Ternary Bonsai 8B.

Yeah those are old ones not the current release.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#156

Post by dofm »

hypercube33 wrote: post #143
janalsncm wrote: post #123

Earlier quoted context omitted.

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

More to the argument that we need a model of models - one general one that calls specialists in to do what they are good at and handles that like a foreman for you.

This is somewhat akin to the "one-expert-per-query" solution Apple are using in their small foundation models I think?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#157

Post by dofm »

janalsncm wrote: post #123
SwellJoe wrote: post #59

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the an…

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

FWIW my tests on my little puzzles suggest that it is not better than the Gemma 4 12B on SQL. It really does seem to get quite tangled up on stuff.

PHP/Wordpress code seems OK (better than the Gemma) but it gets stuck in reasoning loops.

Mind you, I am something of a cynic about the underlying 27B dense Qwen; I think the 35B MoE model is often better and it is just so, so much faster.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#158

Post by parsimo2010 »

jedbrooke wrote: post #75
motbus3 wrote: post #46

I need help understanding this. I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other…

from what I understand prismml isn’t doing a quant like normal models where you take a model trained at fp16 and then chop off some bits to reduce vram, but rather they’re training the model natively with 1 bit weights. It’s explained more in the article. They’re also doing some other tricks like a fp16 weight per block of 128 1bit weights to get some more data out of 1 bit weights

They aren’t training at all. They are quantizing existing models, it’s just that the process is different. The 27B uses Qwen3.6 27B as the base model.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#159

Post by gnabgib »

stamps wrote: post #153
gnabgib wrote: post #150

Earlier quoted context omitted.

Pasta should be made of whole durum wheat.. which is 12g/100g. What's this "good" you speak of? There's certainly worse (soft white/red flour with germ/bran removed), but where do we find the +3g? https://en.wikipedia.org/wiki/Pasta

I know felcetti has a few varieties that exceed that number. ~112g for 16g protein.

The brand that uses khorasan wheat in some versions? Yeah they're a little higher (14.7g/100g). We're stretching the definition of pasta now (it's certainly the highest protein grain you can find)

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#160

Post by dofm »

sosodev wrote: post #146
janalsncm wrote: post #123

Earlier quoted context omitted.

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

The Gemma models are so good at vision. It seems particularly important for phones. Also, they write in a much more pleasant manner than Qwen imo.

I absolutely agree that Gemma 4 writes well out of the box. Free of a lot of the standard American model blog spam writing style but a little more fluid than Qwen.
Post reply on HN