Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

deepgrove.ai

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#31

Post by beautiful_apple »

ricardobeat wrote: post #21
beautiful_apple wrote: post #14

A benchmark table comparing to Qwen 3.5 35B-A3B seems strange when Qwen 3.6 35B-A3B has been out for some time and is significantly better. I didn't notice the version difference when first reading the article! So this is a heads up to people like me.

Their main comparison is 1-bit Bonsai 27B (Qwen3.6 27B) which beats A3B anyway.

I'm not sure what you mean.

Looking at the chart on this website, Bonsai Qwen 3.6 27B has a lower average benchmark score than Qwen 3.5 35B-A3B (77.1 vs 82.9)

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#32

Post by HelloUsername »

brainless wrote: post #19
walrus01 wrote: post #13

I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning. "please write 250 words on the etymology and history of the word schlong" https://pastes.io/uhshFgn4 The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish com…

Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.

> Ask models to search on the Internet and answer?

To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#33

Post by sznio »

walrus01 wrote: post #22
brainless wrote: post #19

Earlier quoted context omitted.

Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.

I don't, really, but 20B is also not that small... It's an intentionally weird question to see how confidently incorrect something will be. It certainly writes a plausible sounding explanation that could fool someone for whom English is their 2nd or 3rd language, or is not familiar with specific North American slang. It's also something I've seen has great results with esoteric individual pieces of knowledge that wor…

Parameter count is not everything.

20b parameters * 1.5 bits per parameter is just 30 billion bits, about 3.75gb

a full 20b fp16 is about 40GB.

I find it weird how a smaller model still produces decent text, except it bullshits all the way.

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#34

Post by spider-mario »

brainless wrote: post #19
walrus01 wrote: post #13

I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning. "please write 250 words on the etymology and history of the word schlong" https://pastes.io/uhshFgn4 The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish com…

Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.

Maybe we don’t necessarily expect them to answer from their own knowledge, but to either do that or say “I don’t know”.

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#35

Post by boomlinde »

walrus01 wrote: post #13

I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning. "please write 250 words on the etymology and history of the word schlong" https://pastes.io/uhshFgn4 The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish com…

Seems like less of a problem in smaller models where bullshit tends to become very obvious to anyone with half a clue about the given subject than it is in larger models where the illusion is complete enough that the confidently stated answers are very incorrect in more subtle ways.

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#36

Post by sajithdilshan »

walrus01 wrote: post #13

I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning. "please write 250 words on the etymology and history of the word schlong" https://pastes.io/uhshFgn4 The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish com…

I think for smaller models, they need to be more defensive on unknown information and frontier model level tool calling capabilities.

LLMs are kind of a compact knowledge box of its training data and it's understandable it would not have information about every topic and in that case just do a web search or a proper tool invocation to get the data and then synthesize.

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#37

Post by CTDOCodebases »

HelloUsername wrote: post #32
brainless wrote: post #19

Earlier quoted context omitted.

Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.

> Ask models to search on the Internet and answer? To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.

I thought the benefit of small models is that they are a natural language UI to whatever they are connected to.

It seems strange to me to expect a small model to answer everything correctly when every device that they are being used on support networking and knowledge is constantly evolving.

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#38

Post by hrmon »

walrus01 wrote: post #13

I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning. "please write 250 words on the etymology and history of the word schlong" https://pastes.io/uhshFgn4 The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish com…

English Wikipedia: "As of 16 October 2024, the size of the current version including all articles compressed is about 24.7 GB without media."

Models in this size range should aim for correct tool calling and avoiding hallucinations, not universal knowledge. (apparently they don't.)

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#39

Post by SwellJoe »

The most interesting thing is the "dreaming" idea for on-device adaptation. I guess regularly adjusting weights becomes possible when the model is so small and the math is simplified.

I'm assuming that doesn't actually exist yet, though, as I don't see anything about an implementation in the code that's been released.

But, it's a really interesting idea for a personal model. There's a risk of more AI psychosis if these things actually start "learning", but the value of it is also probably pretty big. I'm not sure I buy it will actually be able to self-improve, though. The best models are helping improve themselves, but the best models are considerably smarter and more capable than this one. I've asked models like Gemma 4 31B to help figure out training and synthesizing data, and it mostly fails on anything more than categorization and summarization. This little model is much dumber than that.

So, I'm skeptical, but maybe there's deterministic tooling that can assist and maybe it will be scoped tightly enough to just learn and update facts and not so much try to retrain the whole thing.

Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone

#40

Post by woadwarrior01 »

getcrunk wrote: post #18

Anyone compare this to ternary bonsai vs the 1 bit

I haven't compared it for perf, but at first glance this appears to be an LLM (text-only) while both binary and ternary bonsai 27B models are VLMs (where the vision tower and the adapter MLP weights are unquantized and kept in float16).
Post reply on HN