Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

181–190 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#181

Post by kennywinker »

mullingitover wrote: post #166
gnabgib wrote: post #150

Earlier quoted context omitted.

Pasta should be made of whole durum wheat.. which is 12g/100g. What's this "good" you speak of? There's certainly worse (soft white/red flour with germ/bran removed), but where do we find the +3g? https://en.wikipedia.org/wiki/Pasta

From your own link: > Main ingredients: Durum wheat flour, water/ eggs May be going out on a limb here but I think it might be the eggs?

What LLM model are you using to figure that out? The reasoning mode must be frontier level.

/satire, obviously ;)

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#182

Post by kaycey2022 »

hypercube33 wrote: post #143
janalsncm wrote: post #123

Earlier quoted context omitted.

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

More to the argument that we need a model of models - one general one that calls specialists in to do what they are good at and handles that like a foreman for you.

Is that different from mixture of experts?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#183

Post by kennywinker »

thomasjb wrote: post #17

I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)

Toss the rtx into a cheapo optiplex or thinkcenter, and run it headless - the load on your machine is gonna make doing other stuff while it’s running painful. Plus that frees up the rest of your vram.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#184

Post by goofy_lemur »

I tried this on M1 Pro today with 16GB ram and it worked!!!

I was using vscode and it seemed to interperet the system prompt right and then started actually inspecting and doing stuff.

Unfortunately the vscode system prompt is 24000 tokens, and I was getting 100 at beginning, 69 by the end of it, but honestly I'm super impressed. Great work team 1

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#185

Post by RandyOrion »

Thanks Bonsai team.

Now open weight LLMs/VLMs/LMMs are becoming even larger to the extent that consumer-grade hardware are no longer able to run these models. In contrast, quantization and pruning make the model better at the size-performance pareto and provide people with strictly more possibilities.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#187

Post by apitman »

digdugdirk wrote: post #61
kamranjon wrote: post #53

After using a highly capable 2-bit quant as my daily driver for months now, I get pretty excited about releases like this. After a few days for the kinks to be worked out, I’ll be excited to try it.

What model? And what hardware do you run it on? I find these style of models are great, but fail hard , and fail randomly. I'd be hesitant to use it for a daily driver, but I'm using dual 3060s, so it's not like I'm quantizing a frontier model here. How do you find the overall experience? And do you have any special sauce or recommendations for going this route?

I've got dual 3060s as well. What's the best models you've found for this setup?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#188

Post by kamranjon »

pylotlight wrote: post #173
kamranjon wrote: post #80

Earlier quoted context omitted.

DeepSeek V4 Flash with DwarfStar: https://github.com/antirez/ds4 The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.

But isn't it running basically 1 request at a time? This would make agentic coding difficult right? Compared to running as many sequential tests as you want via api?

Yea I don’t use sub-agent style workflows. I often use planning patterns and generate markdown for really complex tasks, but never got into the sub-agent thing. I am likely closer to AI-assisted, though I am heavily using pi coding agent and my editor is basically just for viewing files and changes now.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#189

Post by kamranjon »

trvz wrote: post #86

I still don’t see the point of this. In my testing, it’s worse than Qwen 3.5 4B and even 0.8B.

When new models are released (I realized this is qwen 3.6 but the quant is novel) - it takes a few days for the kinks to get worked out - you’ll likely have better luck if you give it a few days and try again.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#190

Post by ArcHound »

kennywinker wrote: post #180
bigmadshoe wrote: post #144

Earlier quoted context omitted.

Bitter lesson is knocking. Mixture of experts is essentially what you’re describing but free from unnecessary inductive biases.

There is value in splitting things. If all I ever do is local app automations, i don’t need model that knows how to code. If all I ever do is coding, i don’t need a model that translates english to slovakian.

Slovakia mentioned, let's gooo. Ehm, exactly, we can achieve better smaller models for specialized tasks rather than using compute to improve a big model that does everything. There's a lingering philosophical question if better language processing capabilities translate to better image processing capabilities (i.e. having the vocabulary and experience to properly describe an image), but I still think that identifying tasks and splitting responsibilities saves a lot of effort.
Post reply on HN