Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

31–40 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#31

Post by petu »

PcChip wrote: post #26
NitpickLawyer wrote: post #9

Earlier quoted context omitted.

There's two variants of this (or, as the joke goes, for very big values of bit): Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. 1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

this is a really dumb question, but how is -1 represented? is it a float? if so, how many bits is the float? I've never heard of a bit ever having more than two possible values

packing multiple trits together

e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#33

Post by verdverm »

luckystarr wrote: post #22

Tried it on Android and got "!!!!!!!!!!!!!" for answers.

That's what happens when you quant too hard. I'm working on quant strats and evals for the same underlying qwen 27b models.

When I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#34

Post by gunalx »

liuliu wrote: post #13
erelong wrote: post #12

I was trying Ornith 9B locally (it's up on Ollama) which claims: > Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B. https://deep-reinforce.com/ornith_1_0.html Only tried it so much so far; it did a little better than Qwen 9B

Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).

3.5 9B can do thinking. Its just disabled by default in its gguf chat template.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#35

Post by verdverm »

erwan577 wrote: post #23

The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows. I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.

quanting kv cache hurts attention / recall, and long-form tasks by proxy. Model families and sizes have different tolerances to quant ting different parts of the model, same for intended tasks.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#36

Post by verdverm »

erelong wrote: post #12

I was trying Ornith 9B locally (it's up on Ollama) which claims: > Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B. https://deep-reinforce.com/ornith_1_0.html Only tried it so much so far; it did a little better than Qwen 9B

Orinth was not impressive in my vibes testing, I just completed my first grid analysis with real evals on qwen 27b. I can now scale that grid analysis and intend to include the qwen 9b ftunes I've seen going around. They were actually a main motivation because so many claim this or that one is better, but very little in the way of evals

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#37

Post by doctoboggan »

trollbridge wrote: post #27
sigbottle wrote: post #16

What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?

If you read to the bottom of the page, it says they're funded by a few people, and one of them is Samsung. I'm betting Samsung wants to be able to ship a capable AI system on a future model of their phone so they can compete with Apple.

Agreed, and the prevailing wisdom now seems to be that unless you can release a truly frontier model, you might as well release yours as open source to undercut your competition.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#38

Post by dakolli »

thomasjb wrote: post #17

I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)

start saving your money.
Post reply on HN