For those curious about their demo, I’m pretty sure it’s using Locally AI (iOS only) that lmstudio acquired/aquihired a couple months ago.
Bonsai 27B: A 27B-Class model that runs on a phone
11–20 of 278 posts
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#12I was trying Ornith 9B locally (it's up on Ollama) which claims:
> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.
https://deep-reinforce.com/ornith_1_0.html
Only tried it so much so far; it did a little better than Qwen 9B
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#13I was trying Ornith 9B locally (it's up on Ollama) which claims: > Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B. https://deep-reinforce.com/ornith_1_0.html Only tried it so much so far; it did a little better than Qwen 9B
Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#14I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#15I was trying Ornith 9B locally (it's up on Ollama) which claims: > Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B. https://deep-reinforce.com/ornith_1_0.html Only tried it so much so far; it did a little better than Qwen 9B
Is that a 1-bit LLM? I don’t understand the connection with this article.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#16What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#17I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#18That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?
Doing some naive math, the F16 filesize is ~53.8gb, the 1-bit version is ~3.8gb, about 7% of the original size. The F16 size is roughly 2x param count, so that gives a rough ballpark of ~110B.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#19This must be some sort of unpublished app? I can just see their image tool on the app store
It's a LLM model, not a phone app.
Available on HuggingFace: https://huggingface.co/collections/prism-ml/bonsai-27b
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#20I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.
Most probably not optimized yet for this model...