Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

101–110 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#101

Post by mandeepj »

segmondy wrote: post #87
rogerkirkness wrote: post #67

Earlier quoted context omitted.

Apple would punish him severely unless they cleared it in advance, it might be to their advantage for some reason (negotiating with Google for Gemma rights? idk).

Apple is too desperate to be making demands. You confuse the Apple of yesterday and of today. Things change fast and things have changed.

> Apple is too desperate to be making demands

They don't give a F about AI or any new AI model that was announced this morning. Wasn't there news a while ago about them buying Perplexity?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#102

Post by b89kim »

liuliu wrote: post #55
gunalx wrote: post #34

Earlier quoted context omitted.

3.5 9B can do thinking. Its just disabled by default in its gguf chat template.

It is disabled because it doesn't work :) Try it and see the doom loop it gets itself in.

I had no failure mode in 3.6 9B thinking with llama.cpp. After release, there were updates for both model and llama.cpp.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#103

Post by mgambati »

conception wrote: post #97
segmondy wrote: post #87

Earlier quoted context omitted.

Apple is too desperate to be making demands. You confuse the Apple of yesterday and of today. Things change fast and things have changed.

You’re right, Apple only has 68 billion dollars in cash, up 40% since last year. Definitely on their last legs.

Free cash? Crazy. I think I can live with the interest of that in the bank.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#104

Post by thomasjb »

dakolli wrote: post #38
thomasjb wrote: post #17

I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)

start saving your money.

Or watch and wait as models get denser

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#105

Post by dofm »

verdverm wrote: post #70
SwellJoe wrote: post #59

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the an…

4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn. Good evaluation from 2024 https://arxiv.org/pdf/2402.18158 I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I…

The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better).

The 12B QAT model is indeed sort of mindblowing.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#107

Post by verdverm »

dofm wrote: post #105
verdverm wrote: post #70

Earlier quoted context omitted.

4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn. Good evaluation from 2024 https://arxiv.org/pdf/2402.18158 I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I…

The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better). The 12B QAT model is indeed sort of mindblowing.

I haven't dug into QAT deeply, better recovery is my understanding as well, and also that it is out of reach for most people because you have to train a model to back prop errors based on estimated error under quant.

Hopefully more of the lab releases are trained under QAT so we can all benefit.

Post reply on HN