Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bonsai 27B: A 27B-Class model that runs on a phone

prismml.com

Post reply on HN
Print view View original

201–210 of 278 posts

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#201

Post by moffkalast »

janalsncm wrote: post #123
SwellJoe wrote: post #59

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the an…

Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.

> slightly worse on knowledge and tool calling

Worse than Gemma at tool calling? Gemma's already bottom tier at that (at least when there's Qwen to compare to), that would just be unable to do tool calling at all.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#203

Post by timfsu »

sandworm101 wrote: post #120
RugnirViking wrote: post #106

maybe its nitpicking here but the demo shows them asking the model what to cook and its recipie sounds like it wouldn't be very good and also it totally gets the macronutrients wrong. 25g protein for "spaghetti, carrots, peppers, garlic and herbs"?

And why would i want such mundane questions to be handled by an AI on my phone? That sort of thing doesnt need AI, let alone a local one. Basic google search was answering those questions long ago. My point: phone-sized AI is only useful if it can do things that only AI can do. Can it ingest a document scanned by the phones camera? Can it translate in real time? I dont see how or why i would ever ask it for recipe ad…

I had exactly this use case - in a grocery store in the Alps, no internet, fired up a local LLM on my phone to figure out what to cook and what to buy

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#204

Post by xyzsparetimexyz »

kennywinker wrote: post #183
thomasjb wrote: post #17

I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)

Toss the rtx into a cheapo optiplex or thinkcenter, and run it headless - the load on your machine is gonna make doing other stuff while it’s running painful. Plus that frees up the rest of your vram.

Why would they do that when they already have a perfectly good PC?

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#206

Post by [deleted] »

purerandomness wrote: post #145
RugnirViking wrote: post #106

maybe its nitpicking here but the demo shows them asking the model what to cook and its recipie sounds like it wouldn't be very good and also it totally gets the macronutrients wrong. 25g protein for "spaghetti, carrots, peppers, garlic and herbs"?

Good spaghetti has 15% protein, 15g per 100g. Even the lowest quality one has 12g/100g.

[deleted]

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#208

Post by networked »

I have benchmarked Bonsai 27B CPU inference on my computer (a Ryzen 7 5700X desktop with 48G RAM running Ubuntu 24.04) using the latest 62061f910 build of PrismML's llama.cpp fork.

Binary: 9 t/s prompt, 6 t/s generation. Ternary: 0.8 t/s prompt, 0.7 t/s generation. It looks like CPU inference for ternary isn't optimized yet.

Re: Bonsai 27B: A 27B-Class model that runs on a phone

#209

Post by kennywinker »

xyzsparetimexyz wrote: post #204
kennywinker wrote: post #183

Earlier quoted context omitted.

Toss the rtx into a cheapo optiplex or thinkcenter, and run it headless - the load on your machine is gonna make doing other stuff while it’s running painful. Plus that frees up the rest of your vram.

Why would they do that when they already have a perfectly good PC?

…? I said why.

> the load on your machine is gonna make doing other stuff while it’s running painful

Is your question about something else?

Post reply on HN