My question is what LLM the author used for dev the web page? Maybe Claude? or ChatGPT?
Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
51–59 of 59 posts
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#52Earlier quoted context omitted.
I think for smaller models, they need to be more defensive on unknown information and frontier model level tool calling capabilities. LLMs are kind of a compact knowledge box of its training data and it's understandable it would not have information about every topic and in that case just do a web search or a proper tool invocation to get the data and then synthesize.
You need a small LLM that can reason very well and use tools like web search very well. No one is going to compress human knowledge into few bits.
Not at the moment, but who knows what types of models and storage types we would have in 100 years.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#53I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning. "please write 250 words on the etymology and history of the word schlong" https://pastes.io/uhshFgn4 The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish com…
They are not incorrect. They don't even hold a position in the first place.
An LLM continues the prompt it is given. What is more likely to come after a question? An answer, not an "oh sorry I'm not sure". Sure, you could make the latter more likely, but then the model would be unusable. Larger models simply contain more answers, more ways to stumble into them, and a granular enough geography to stay on the trail.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#54Earlier quoted context omitted.
You need a small LLM that can reason very well and use tools like web search very well. No one is going to compress human knowledge into few bits.
Not at the moment, but who knows what types of models and storage types we would have in 100 years.
It's far more likely basic consumer devices will advance such that much larger models efficiently run than finding novel ways to compress all of human experience to fit on today's mobile hardware.
Information can only be compressed so much
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#55Earlier quoted context omitted.
Not at the moment, but who knows what types of models and storage types we would have in 100 years.
It's far more likely basic consumer devices will advance such that much larger models efficiently run than finding novel ways to compress all of human experience to fit on today's mobile hardware. Information can only be compressed so much
We’re already seeing incredible knowledge compression out of the models the GP mentioned like Qwen 35B-A3B, which feels well within the realm of “runs on a phone” in the next handful of years.
And by then we’ll probably have been further surprised by just how much information and capacity for reasoning can be crammed into a few gigs of weights. Models just keep getting better for a given size, it’ll be interesting to see where the limit of that is.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#56Earlier quoted context omitted.
Their main comparison is 1-bit Bonsai 27B (Qwen3.6 27B) which beats A3B anyway.
I'm not sure what you mean. Looking at the chart on this website, Bonsai Qwen 3.6 27B has a lower average benchmark score than Qwen 3.5 35B-A3B (77.1 vs 82.9)
The 27B model beats 35B A3B. The quantized 1-bit version obviously doesn't, but it would likely beat a 1-bit quantized A3B variant. Comparing 3.5 35B or 3.6 35B to this model (Maple) is not meaningful.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#57>"At ultra-low bitwidths,
matrix multiplication can be effectively replaced with additions
, lowering the total arithmetic workload needed to infer through a model."
Interesting! I never knew that before... I will have to do some more research on this, but yes, it definitely sounds possible!
(Also (and this is just a wild guess/hunch here!)
in theory, matrix multiplication could be replaced with pre-computed cached lookups from a lookup table in memory
if, if and only if the resultant pre-computed lookup table of result matrices could fit into memory available!
In other words, think of a Matrix not as a 2D array, but as a 1D string. Think of the Matrix multiply operation as first appending the 1D string of the first matrix to the 1D string of the second matrix, then using that string as a Key to look-up a precomputed result in Memory...
Of course, the limitation to this approach is the availability of memory relative to the permutations of input values... It could work -- but only for smaller matrices... which leads to another strange idea... in that set of pre-computed matrix permutations, there might be duplicates... figure out a way to put duplicates in memory only once (could use pointers or some other scheme to do this) and you could theoretically store more matrices in the same memory and/or slightly larger matrices!
Anyway, just rambling out loud! :-)
Remember, "almost all programming can be viewed as an exercise in caching" to quote the legendary programmer Terje Mathisen! :-) )
Anyway, great article!
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#58I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning. "please write 250 words on the etymology and history of the word schlong" https://pastes.io/uhshFgn4 The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish com…
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
It would indeed be better to have them search the internet for an answer, but the issue (as I see it) is overconfident models don't know when they should search the internet because they think they already know the answer.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#59looks good