Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...
Bonsai 27B: A 27B-Class model that runs on a phone
21–30 of 278 posts
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#22Tried it on Android and got "!!!!!!!!!!!!!" for answers.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#23The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows.
I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#24I was trying Ornith 9B locally (it's up on Ollama) which claims: > Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B. https://deep-reinforce.com/ornith_1_0.html Only tried it so much so far; it did a little better than Qwen 9B
Is that a 1-bit LLM? I don’t understand the connection with this article.
Oh, I don't actually know the difference if you want to explain it
The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?
edit: I asked AI for the difference and understand a little better, thanks for the heads up to learn the difference between models... I think the thing was, although ornith was created for a specific agentic purpose, it was still outperforming a previous generalist model I had running locally (so in my mind I thought it was still a better local model) - I'd like to try bonsai out if I can figure out how to run it lol
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#25[dead]
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#26TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
There's two variants of this (or, as the joke goes, for very big values of bit): Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. 1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
this is a really dumb question, but how is -1 represented?
is it a float? if so, how many bits is the float?
I've never heard of a bit ever having more than two possible values
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#27What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?
If you read to the bottom of the page, it says they're funded by a few people, and one of them is Samsung. I'm betting Samsung wants to be able to ship a capable AI system on a future model of their phone so they can compete with Apple.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#28The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#29The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
Depending on which model you're running, you might need to use the custom forks.
Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....
Re: Bonsai 27B: A 27B-Class model that runs on a phone
#30Earlier quoted context omitted.
There's two variants of this (or, as the joke goes, for very big values of bit): Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. 1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
this is a really dumb question, but how is -1 represented? is it a float? if so, how many bits is the float? I've never heard of a bit ever having more than two possible values
It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.