Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

231–240 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#231

Post by latexr »

The meal demo is hilarious.

— Hey, model, see this fake-ass stock photo of a variety of spices, vegetables, and spaghetti? What meal can I make with this?

— Just cook everything.

— I’m a complete noob. I can’t even fathom how to cook those things. Help me!

— Sure sure. First boil the spaghetti completely and drain. Only after that, while it’s getting cold, you need to sauté (good luck knowing what that is if you don’t even know how to cook spaghetti) the garlic and carrots at this specific temperature (good luck figuring out how to do that on a stove). Despite having mentioned the peppers and herbs in the previous message, I’m not going to tell you what to do with those. Just chew them raw or something, I guess.

The demo shows that the model can answer, but the answers are frankly bad. Here’s what you could’ve done instead faster with better results: a web search for “spaghetti carrots peppers”. Don’t even need to add “recipe”.

Presumably you’ve been using the model as you develop it, why not show something real and useful instead of a generic, unrealistic and uninteresting scenario that above all makes it look incompetent? Show something that genuinely surprised you positively.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#232

Post by HalfCrimp »

kennywinker wrote: post #183
thomasjb wrote: post #17

I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)

Toss the rtx into a cheapo optiplex or thinkcenter, and run it headless - the load on your machine is gonna make doing other stuff while it’s running painful. Plus that frees up the rest of your vram.

Running an llm on a PC running a desktop environment really isn't that bad. You lose a bit of ram & vram but that only matters if you _reaaaally_ want to push to the max model size your hardware can handle.

The biggest issue I've found is absent mindedly opening YouTube or the like that spike ram requirements and freezing the system up. But that's a me problem

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#233

Post by graysonk »

recursivegirth wrote: post #133
janalsncm wrote: post #123

Earlier quoted context omitted.

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

I think this is where leveraging classifier models will become important. The frontier LLM models do "everything", while we've known for a while that to truly scale this we will need to distill models into their individual functions. I don't see this as necessarily a bad thing and hope more is done in this space. Very promising.

I enjoy doing local image generation and this is one thing that the community around that has really optimized.

In some workflows you might have 20 different models doing their specialized tasks. Pose detection, hand/eye/face detailers, classifiers, refiners, up scalers, taggers, etc can all use their own models and that’s not even the including the model(s) used for the actual image generation part.

I’m interested to see the optimization when this concept gets applied to other general ai tasks.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#234

Post by dofm »

syntaxing wrote: post #14

I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.

It’s still a dense model so all 27B parameters are active. 35B-A3B activates only 3B at any time.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#235

Post by kbart »

Excuse my likely stupid question, but has anybody had some success using Claude Code with frontier agents (or Junie or anything else) to invoke local LLMs for specific sub-tasks or wrapped as skills? In other words, is there a way to use expensive, frontier models as orchestrators that manage local models to do the specialised coding tasks?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#236

Post by adam_patarino »

kbart wrote: post #235

Excuse my likely stupid question, but has anybody had some success using Claude Code with frontier agents (or Junie or anything else) to invoke local LLMs for specific sub-tasks or wrapped as skills? In other words, is there a way to use expensive, frontier models as orchestrators that manage local models to do the specialised coding tasks?

What for? Cost savings? Something else?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#237

Post by lta »

dofm wrote: post #154
trollbridge wrote: post #28

Earlier quoted context omitted.

Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.

Have Prism-ML upstreamed their forked code?

Yeah it's a llama.cpp fork and it's linked everywhere on their repo

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#238

Post by docheinestages »

kbart wrote: post #235

Excuse my likely stupid question, but has anybody had some success using Claude Code with frontier agents (or Junie or anything else) to invoke local LLMs for specific sub-tasks or wrapped as skills? In other words, is there a way to use expensive, frontier models as orchestrators that manage local models to do the specialised coding tasks?

Look into FastContext by Microsoft. Not extraordinary but specifically designed for paired usage with a larger LLM to save tokens [1].

[1]: https://github.com/microsoft/fastcontext

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#239

Post by fabioz »

kbart wrote: post #235

Excuse my likely stupid question, but has anybody had some success using Claude Code with frontier agents (or Junie or anything else) to invoke local LLMs for specific sub-tasks or wrapped as skills? In other words, is there a way to use expensive, frontier models as orchestrators that manage local models to do the specialised coding tasks?

I'm working on the area in https://beolis.com.

The system as a whole is meant to support that use case, where each task (ticket in its jargon) can be tackled using a custom workflow that can each use a different agent/llm (so, it should support local LLMs if you have configured your coding agent to use them).

Sidenote: it's still not where I want, but getting there...

Post reply on HN