Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

251–260 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#251

Post by newswasboring »

rtcoms wrote: post #249
newswasboring wrote: post #241

Earlier quoted context omitted.

That is my entire workflow in opencode through delegates. I have an "orchestrator" which uses matt pocock skills like grilling and writes specs in gh tickets. Once that is done it delegates individual tickets to deepseek v4 flash based developer subagent which executes the code pretty fast. This agent can only delegate one subagent which is a code-reviewer which uses a better model like qwen 3.7 for review. So dev do…

Do you have source code or config for this setup ?

It is just basic agent setup. When I get home in the evening I can isolate my agent files and share GitHub link. In a couple of hours.

Edit: So life got in the way and I couldn't isolate it. But I have made this project public and committed the dev and code review agents here.

https://github.com/rick2047/meuseum-game-new/tree/main/.open...

The delegate builder is too ugly to share yet. But to be honest just use the normal builder with a frontier model and instruct it to delegate to developer any developer task. That works well.

Of course you have to install the skills [1]

[1] https://github.com/mattpocock/skills

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#252

Post by verdverm »

kamranjon wrote: post #192
verdverm wrote: post #62

Preliminary analysis via lm-evaluation-harness + vllm model | disk | wikitext | gsm8k (match/error) baseline | 55G | 8.00 | 0.50/0.09 nvfp4-gptq | 27G | 8.25 | 0.47/0.9 nvfp4a16-gptq | 27G | 8.11 | 0.53/0.9 bonsai-4bit | 19G | 16.75 | 0/0 (eval bug?) Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. I'm also not sure what is up with the gsm8k, their benchmarks show so…

Doesn’t this suggest you aren’t properly running the model?

Its unclear if its in vllm, the eval harness, or the weights. I'm running it the same as other qwen3.6 derived models and it appears to work, but other comments speak of waiting for support in their tool. Could be a chat template or could be legit because I'm using a different quant / bit selection from their hugging face and using their primary model as the comparison point for my huh? They may have put more effort into the flagship model than the 4bit because they are focused on on-device model running.

Software was already complex, now we are adding highly non deterministic elements into the mix

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#253

Post by rtcoms »

newswasboring wrote: post #251
rtcoms wrote: post #249

Earlier quoted context omitted.

Do you have source code or config for this setup ?

It is just basic agent setup. When I get home in the evening I can isolate my agent files and share GitHub link. In a couple of hours. Edit: So life got in the way and I couldn't isolate it. But I have made this project public and committed the dev and code review agents here. https://github.com/rick2047/meuseum-game-new/tree/main/.open... The delegate builder is too ugly to share yet. But to be honest just use the n…

Sure, that would be very helpful.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#254

Post by HnUser12 »

docheinestages wrote: post #238
kbart wrote: post #235

Excuse my likely stupid question, but has anybody had some success using Claude Code with frontier agents (or Junie or anything else) to invoke local LLMs for specific sub-tasks or wrapped as skills? In other words, is there a way to use expensive, frontier models as orchestrators that manage local models to do the specialised coding tasks?

Look into FastContext by Microsoft. Not extraordinary but specifically designed for paired usage with a larger LLM to save tokens [1]. [1]: https://github.com/microsoft/fastcontext

That repo suddenly seems to have gone missing. I get 404.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#255

Post by newswasboring »

rtcoms wrote: post #253
newswasboring wrote: post #251

Earlier quoted context omitted.

It is just basic agent setup. When I get home in the evening I can isolate my agent files and share GitHub link. In a couple of hours. Edit: So life got in the way and I couldn't isolate it. But I have made this project public and committed the dev and code review agents here. https://github.com/rick2047/meuseum-game-new/tree/main/.open... The delegate builder is too ugly to share yet. But to be honest just use the n…

Sure, that would be very helpful.

I edited the comment above

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#256

Post by dofm »

lta wrote: post #237
dofm wrote: post #154

Earlier quoted context omitted.

Have Prism-ML upstreamed their forked code?

Yeah it's a llama.cpp fork and it's linked everywhere on their repo

Of course — I compiled it and ran it yesterday. Not aware if they have a PR in place with the main project, though as I recall I ran the last Bonsai model without needing it so I guess there might be plans.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#257

Post by [deleted] »

segmondy wrote: post #116
mandeepj wrote: post #101

Earlier quoted context omitted.

> Apple is too desperate to be making demands They don't give a F about AI or any new AI model that was announced this morning. Wasn't there news a while ago about them buying Perplexity?

They do, they are low key panicking.

[deleted]

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#258

Post by stiltzkin »

diseasedyak wrote: post #246
kbart wrote: post #235

Excuse my likely stupid question, but has anybody had some success using Claude Code with frontier agents (or Junie or anything else) to invoke local LLMs for specific sub-tasks or wrapped as skills? In other words, is there a way to use expensive, frontier models as orchestrators that manage local models to do the specialised coding tasks?

This may not be exactly what you're asking, but I've been running Hermes using different frontier models (like DeepseekV4-Pro, GLM-5.2, and others) as the heavy lifters but who also spawn agents to run my local models (mainly Qwen3.6-IQ4) for offloading tasks that are a good fit for a quantized model that's local. Works really well.

[dead]

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#259

Post by docheinestages »

HnUser12 wrote: post #254
docheinestages wrote: post #238

Earlier quoted context omitted.

Look into FastContext by Microsoft. Not extraordinary but specifically designed for paired usage with a larger LLM to save tokens [1]. [1]: https://github.com/microsoft/fastcontext

That repo suddenly seems to have gone missing. I get 404.

Oh, I just realized Microsoft removed it a couple weeks ago. I had the link in my bookmarks. The model is available on HuggingFace [1].

[1]: https://huggingface.co/models?sort=trending&search=fastconte...

Post reply on HN