the dreaming demo peaks at 5.9 GB, is that mac-only? ios would jetsam a third-party app well before that on most iphones
Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
41–50 of 59 posts
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#42amazing work! impressed with https://deepgrove.ai/chat
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#43Earlier quoted context omitted.
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
> Ask models to search on the Internet and answer? To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
This is a very bad way of thinking of it. Small LLMs have clues about real knowledge but only surface level answers will be accurate.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#44Earlier quoted context omitted.
these tools are for manipulating and retrieving and transforming sequences of context. using it as a knowledgebase is just expecting the wrong thing.
Sure, then you should allow that they are also tools for transforming sequences of questions into answers. Language models are based on compression of information, using them as a knowledge base is entirely within capability.
> Language models are based on compression of information, using them as a knowledge base is entirely within capability
It's also the aspect of LLMs that degrades fastest with quantisation. You can't reasonably expect accurate knowledge of everything in the world in a few gigabytes.
Which means that small models need to be conditioned to rely more heavily on tools to fetch accurate information, and ideally not try and generate facts purely based on their (extremely lossy) internal knowledge
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#45Earlier quoted context omitted.
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
> Ask models to search on the Internet and answer? To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
The real target of small models is surely the opposite: knowing with clarity how to conversationally or otherwise reason about your requirements in natural language, sound and imagery, consult APIs that interact with apps, and then have a strong, generic capability to reason about what it finds.
If you want it to know more things locally, you could hopefully distribute wikipedia or other appropriate large sources on a flash storage, properly pre-indexed, with a local tool API, and let it work its magic on that. Not least because that is as I understand it, considerably easier to incrementally update.
This is, without doubt, a very difficult target, not least because it is rather at odds with the way LLMs work on a crude level.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#46Earlier quoted context omitted.
Their main comparison is 1-bit Bonsai 27B (Qwen3.6 27B) which beats A3B anyway.
Beats how? In my experience 1-bit bonsai 27B is quite "dumb" when asked a question about a lot of things, as a canned repository of static knowledge from its training dataset. I mean, I literally asked it for a 250 word description of Seattle and it hallucinated a tallest building in the city with an observation deck that doesn't exist, and didn't mention the Space Needle. Really basic stuff. But then again, the enti…
I also find that the 35B A3B in Q4 outdoes Ternary Bonsai 27B.
Mind you, because I am not that interested in long-context, fire-and-forget tasks, I think the faster MoE model has more value generally.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#47Earlier quoted context omitted.
these tools are for manipulating and retrieving and transforming sequences of context. using it as a knowledgebase is just expecting the wrong thing.
Sure, then you should allow that they are also tools for transforming sequences of questions into answers. Language models are based on compression of information, using them as a knowledge base is entirely within capability.
It's kinda like lossily compressing the entire content of a CD-DA disc to fit onto a 3.5" floppy disk and then complaining that it doesn't sound very nice.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#48Earlier quoted context omitted.
Would it not be better to ask models to search the topic on the Internet and then answer? I do not understand why we expect small LLMs to answer from own knowledge.
> Ask models to search on the Internet and answer? To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
True but they could query a local 50GB Wikipedia dump for factual info’s via embedding search. All still local.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#49Earlier quoted context omitted.
> Ask models to search on the Internet and answer? To me, the benefit of running small models is that they fit on your device exactly that you don't need any internet connection. It's all local and offline, so you can still consult for information in any scenario.
True but they could query a local 50GB Wikipedia dump for factual info’s via embedding search. All still local.
I wonder if someone already made a somewhat turnkey setup like this? Small models should ideally not waste a lot of parameters on trying to memorize too much knowledge and then hallucinate / interpolate a lot of the gaps.
Re: Show HN: Maple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhone
#50I wish that "small" LLMs would stop being confidently very incorrect. Admittedly this is a bit of an intentionally esoteric test, but the confident way in which it presents a totally incorrect answer is a bit concerning. "please write 250 words on the etymology and history of the word schlong" https://pastes.io/uhshFgn4 The actual origin of the word is from middle high German and Yiddish-speaking Ashkenazi Jewish com…
I think for smaller models, they need to be more defensive on unknown information and frontier model level tool calling capabilities. LLMs are kind of a compact knowledge box of its training data and it's understandable it would not have information about every topic and in that case just do a web search or a proper tool invocation to get the data and then synthesize.
You need a small LLM that can reason very well and use tools like web search very well.
No one is going to compress human knowledge into few bits.