Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

71–80 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#71

Post by JoshTriplett »

petu wrote: post #31
PcChip wrote: post #26

Earlier quoted context omitted.

this is a really dumb question, but how is -1 represented? is it a float? if so, how many bits is the float? I've never heard of a bit ever having more than two possible values

packing multiple trits together e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing

It's impressive how close to optimal this is.

You can beat the efficiency of 5 trits in 8 bits (1.6) with as few as 17 trits in 27 bits (~1.588), but once you account for rounding up to a whole number of bytes for practical reasons, then beating the efficiency requires going to at least 111 trits in 176 bits (~1.586), or perhaps more practically for fast unpacking, 161 trits in 256 bits (~1.59).

At that level, even if you have, say, 27B trits, the more efficient encodings would save something like 38-45MB (theoretical limit ~48MB), likely at the cost of some slowdown.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#73

Post by drob518 »

xyzsparetimexyz wrote: post #10

That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?

Yep, that’s the question. I asked just that when Bonsai’s first models got released. Super interesting if we can push the parameter count over 100B with 1.125 bit quantization and still keep pretty good performance versus 16-bit 100B models. That’s a definite sweet spot.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#74

Post by drob518 »

verdverm wrote: post #36
erelong wrote: post #12

I was trying Ornith 9B locally (it's up on Ollama) which claims: > Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B. https://deep-reinforce.com/ornith_1_0.html Only tried it so much so far; it did a little better than Qwen 9B

Orinth was not impressive in my vibes testing, I just completed my first grid analysis with real evals on qwen 27b. I can now scale that grid analysis and intend to include the qwen 9b ftunes I've seen going around. They were actually a main motivation because so many claim this or that one is better, but very little in the way of evals

I tried it, too, and it got stuck in some loops where it couldn’t recover. Shame, it was promising for the same reason as Bonsai’s models.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#75

Post by jedbrooke »

motbus3 wrote: post #46

I need help understanding this. I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other…

from what I understand prismml isn’t doing a quant like normal models where you take a model trained at fp16 and then chop off some bits to reduce vram, but rather they’re training the model natively with 1 bit weights. It’s explained more in the article. They’re also doing some other tricks like a fp16 weight per block of 128 1bit weights to get some more data out of 1 bit weights

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#77

Post by maxvu »

CharlesW wrote: post #58
kristianp wrote: post #21

Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...

Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.

The 2000 ATI leak showed up here a few weeks ago.

https://www.theregister.com/on-prem/2000/08/02/jobs-snubs-at...

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#78

Post by mchusma »

Onavo wrote: post #45
oceansweep wrote: post #41

Earlier quoted context omitted.

Why make this comment without having tried it first? It very clearly is not useless and performs a lot better than one might expect. I am currently waiting to do more benchmarks of it in comparison to the full weight model, but it seems promising/better than Mistral Nemo at a lower file size.

I think what OP means is that the "minimum viable product" for a daily use LLM is probably somewhere around e.g. GPT 4o's level of intelligence (YMMV). Below a certain threshold, you are better off using specialized machine learning models rather than general purpose LLMs. It's very difficult to get that level of intelligence fully local on a mobile device without streaming to the cloud.

I do think this would be interesting if they made these easy to finetune, as I do think this level of intelligence is likely sufficient for many applications and could be extremely cheap to run.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#79

Post by kamranjon »

digdugdirk wrote: post #61
kamranjon wrote: post #53

After using a highly capable 2-bit quant as my daily driver for months now, I get pretty excited about releases like this. After a few days for the kinks to be worked out, I’ll be excited to try it.

What model? And what hardware do you run it on? I find these style of models are great, but fail hard , and fail randomly. I'd be hesitant to use it for a daily driver, but I'm using dual 3060s, so it's not like I'm quantizing a frontier model here. How do you find the overall experience? And do you have any special sauce or recommendations for going this route?

I’m using DeepSeek V4 Flash on 128gb mbp - it’s a bit different using a 200b+ param model. It’s MoE so performance is acceptable. It will still malform a tool call every now and then, but the capabilities are so far ahead anything else that the majority of the time it works really well and solves really complex problems.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#80

Post by kamranjon »

RALaBarge wrote: post #68
kamranjon wrote: post #53

After using a highly capable 2-bit quant as my daily driver for months now, I get pretty excited about releases like this. After a few days for the kinks to be worked out, I’ll be excited to try it.

What have you been using?

DeepSeek V4 Flash with DwarfStar: https://github.com/antirez/ds4

The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.

Post reply on HN