181–188 of 188 posts
It seems to fail. I sent the prompt: “ 5° warmer” And it said: “ setting the temperature to 5°F”
Either way, setting temperature to 5° is obviously wrong - even if it knew the current temperature - but models of this size can't reason about relative values very well.
Give it a tool to change temperature by a given amount, and see what happens!
Earlier quoted context omitted.
Tested your example, the confidence is 0. In smart home context, I can think of an application where the low confidence answers can be forwarded to cloud, whereas the vast majority generic queries solved locally, if the confidence is reliable enough. The response is quite fast by the way.
But this isn’t a query that should need to be forwarded to the cloud for acting on!
It cannot be used for direct customer interaction, it's simply too limited.
Nobody wants to say "lock the front door", they would just do it
And even quite direct commands like "tighten security" are not understood by the model
This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack. With that being said, the web demo is not particular…
For one thing, on beefy-enough recent CPUs, you could keep the weights hot in the L2 cache of a single CPU core. (Which is clearly not the use-case, but might be interesting to those looking for extreme TPS numbers. Or perhaps for efficient training!)
I’d be interested in attempting to run this in a network- enabled 32MiB RAM microVM.
Then I tried using it on a raspberry pi 4. Did not work.
This is specifically for 64 bit arm. I see no reason why you could not enable desktops as well? Just a matter of compilation