Earlier quoted context omitted.
> I love the idea of an on-device model My impression is that most people here haven't tried similar small models and don't have first hand experience with them. They are, to be honest, terrible. They may be good for certain tasks, but are much weaker than something like GPT4. I don't feel excited about these small models that are not fast yet hallucinate all the time.
Weaker by what metric? Are you asking them to explain the fall of Rome to you? The point of a small model isn't to be an interactive Wikipedia. It's there to call tools, get more data, aggregate the data and return a natural language result. It does not "hallucinate", because it only uses what the tools provide.
I'm not going to use any small model that has a chance of dialing 911 when asked to send a text message to a friend.