Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

cactuscompute.com

Post reply on HN
Print view View original

101–110 of 181 posts

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#101

Post by tomrod »

ehnto wrote: post #100
tomrod wrote: post #93

Earlier quoted context omitted.

I confused by the dispatch model. Tool calls typically need some reasonability to be deterministic and, more importantly, predictable in response (o/w GIGO). Why would the thermostat need to interpret a voice command that the node capturing the voice command would not interpret instead?

The node capturing the voice command could be the thermostat. From my understanding they are targeting very small devices. So that could be a master home automation node, but why not also a single purpose device? I can think of more bad examples than I can good ones, but maybe I am doing some soldering and I need my soldering iron turned up a bit; my hands are full, so doing that by voice would be useful enough. Some…

Oh I agree that's not unreasonable. I wonder about the harnessing heft required to make it feasible though. If I say I'd like it a bit warmer, a tool can deterministically bump a few degrees while an LLM might bump it 10C. So do we limit the tool's range?

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#103

Post by justoneletter »

petu wrote: post #13
Tiberium wrote: post #4

Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so: Query: HN Result: { "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 } I'd expect it to at least ignore (call no tools) for the queries that it doe…

"confidence": 0, so I guess you could threshold it

Only if the confidences are calibrated, and they probably aren't. Any idea how the confidence is derived?

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#104

Post by anon373839 »

HenryNdubuaku wrote: post #33
nater5000 wrote: post #24

This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack. With that being said, the web demo is not particular…

So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.

It seems to me that the model struggles to have enough general intelligence, knowledge, or reasoning capacity for arbitrary prompted tool calling. At this size, not surprising.

I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#106

Post by silentbob7 »

nater5000 wrote: post #24

This is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack. With that being said, the web demo is not particular…

I'm quite impressed by the results of the web demo, especially given its size and the precision with which it uses the three available tools (tested with German commands). I could imagine that this LLM would fit well into a setup with multiple micro-sized LLMs for different purposes; so 14 MB for precise tool invocation is a reasonable memory footprint when a number of other local models are running (e.g. STT, TTS + language models).

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#107

Post by peter_d_sherman »

Utterly Fascinating!

For the longest time, I conceptualized LLM's as Text Input -> Text Output transformers, then later as Text Input -> Video Output transformers. Later still I conceptualized them (if they were general purpose) as Any Format Input -> Any Format Output transformers...

The idea of a smaller parameter model runable on smaller/slower/less complex hardware (computers with no GPU, slower CPU's, less memory, aka "Edge Devices") trained for Text Input -> JSON Output (used for tool calls, etc.) I could honestly not conceptualize before seeing the demo on the web page...

But now that I've seen it and conceptualized it -- I'd have to say: "Yes, there's definitely a huge niche, a huge market for this, directly between the non-LLM driven tools and software and SaaS's of yesteryear, and the latest, cutting edge Frontier AI models of today!"

So, I like Needle a lot!

I like Needle a lot, and I love the idea of any tiny resource-thrifty LLM that can run on older hardware, that outputs only JSON!

I can see a huge market for it!

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#108

Post by silentbob7 »

nater5000 wrote: post #19
dofm wrote: post #10

Naïve and clumsy question: how would you pair this with speech-text-speech stuff, wake words etc.? Are there good examples of this for a Pi 5? The demo is super — I'm just having trouble seeing the whole picture for e.g. a screenless device. ETA: pun not intended

The best entrypoint is Home Assistant: https://www.home-assistant.io/ That will get you a lot further than what you're asking, but if you dig a bit through Home Assistant features, resources, etc., you may find the current "best" answers to your questions. If you want a quick answer: Whisper is a good open-source speech-to-text model which comes in a variety of sizes ( https://huggingface.co/openai/whisper-tiny ). Yo…

The wyoming protocoll seems to be the path for home assistant audio, so you need STT (wyoming-faster-whisper), TTS (wyoming-piper for wide language support) API endpoints and some Ollama or OpenAI API endpoint available for your home assistant server.

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#109

Post by planb »

yoavm wrote: post #8
Tiberium wrote: post #4

Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so: Query: HN Result: { "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 } I'd expect it to at least ignore (call no tools) for the queries that it doe…

The website says the model is for "tool calling, device use, and structured extraction". Your example just doesn't seem to be very relevant. FWIW, it did a pretty good job for tool calling when I tried it, and I think it could be pretty nice to have this running on locally and integrate with Home Assistant.

This is extremely impressive if it works. But on the other hand, if the number of cases where it works as expected is lower than what we could catch with a (old-Siri-style) heuristics based approach, and the rest fails in unpredictable ways, I'd prefer the dumb old "I did not understand that" response.

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#110

Post by silentbob7 »

yorwba wrote: post #51

"make it as dark as possible" { "function_calls": [ { "name": "set_thermostat", "arguments": { "temperature": 72, "mode": "cool", "room": "living room" } } ], "reasoning": "'as dark as possible' -> set_thermostat to warm; 'dark' implies higher temperature; 'cool' mode for darkness.", "confidence": 0 } ... maybe this counts as dark humor at least. Since it seems limited to matching a few templates and otherwise fallin…

I also wonder how small a LLM trained on catching only subject (e.g. living room) and action (light on) from text input could be compared to needle - the json wrapping could be done afterwards using templates.
Post reply on HN