What is the difference between this and random sentence generator?
Ask it to lock a door for instance. It seems to convert simple instructions to reasonable tool calls. Check its confidence score.
Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
21–30 of 188 posts
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#22Was really cool to see yous use Engrams to cut down compute! Given its basically an O(1) lookup with disk space being the main constraint, I was curious if you've tried ablating engram layers and sizes across your setup? Also, why mHC over attention residuals?
Yes, we ablated Engrams rigorously and found that it returned world knowledge like FFN without without compute expenditure.
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#23Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so: Query: HN Result: { "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 } I'd expect it to at least ignore (call no tools) for the queries that it doe…
This is exactly why the confidence feature was introduced, the model knows when its wrong, we could hide that part and return a placeholder "sorry I only do function calls", would that be better or you prefer to see everything?
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#24This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#25The learned confidence gate is the crucial piece for a 14MB action model. On ambiguous requests such as the HN example, what calibration target decides between abstaining locally and escalating to the cloud?
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#26This is cool!
While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).
Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#27I’d be interested in attempting to run this in a network- enabled 32MiB RAM microVM.
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#28Earlier quoted context omitted.
False positives are definitely relevant and worth measuring - natural language interfaces always have a discoverability problem, i.e., users not knowing what actions the system does and does not support. If the frontend of that system lacks the ability to reject unsupported commands, weird stuff happens. Nonetheless, this is very cool work! If I can offer a small suggestion to the team at Cactus, it would be to evalu…
> "confidence": 0 OP and the linked page talk about the confidence score and using it as an action threshold, so it looks like an appropriate total response to me.
Right, but that's not the same thing as reporting a benchmark across a test set. It doesn't help me determine how well the model does across a decently-large sample size of commands. It doesn't tell me with what reliability the confidence will be below a given threshold when it should be, above that threshold when it should be, etc.
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#29I’d be interested in attempting to run this in a network- enabled 32MiB RAM microVM.
Thanks, how can we help?
Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots
#30This is cool! While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form). Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.
thanks!