Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

cactuscompute.com

Post reply on HN
Print view View original

161–170 of 181 posts

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#161

Post by HenryNdubuaku »

stavros wrote: post #160
a022311 wrote: post #154

Looks incredibly cool, although it ran at ~4 tok/s on my browser (I assume because I've disabled V8). I believe your numbers though. I always wanted to try using something like this to create an AI assistant for my phone. If you try to create something without AI, you'll quickly run into the problem of using NLP to map input to the equivalent of AI tool calls. On the other hand, shelling out to a full-blown LLM for e…

310 tok/sec on my browser, hm.

Now you can try running the python package directly on your Mac to see proper speedup. WebAssembly has some limitations, running on pure hardware, especially ARM-based to see thousands (which 99% of consumer devices are ARM).

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#162

Post by HenryNdubuaku »

watwut wrote: post #152
ProtoAES256 wrote: post #147

Earlier quoted context omitted.

It's called a LM :). LLM stands for Large Language Model.

LLM as in Little Language Model

Ok we'd be adopting Little Language Model officially haha

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#163

Post by londons_explore »

anon373839 wrote: post #104
HenryNdubuaku wrote: post #33

Earlier quoted context omitted.

So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.

It seems to me that the model struggles to have enough general intelligence, knowledge, or reasoning capacity for arbitrary prompted tool calling. At this size, not surprising. I am VERY interested in seeing how it could perform with some fine-tuning for a specific family of tools/tasks. That would be a great addition to the demo.

It also seems to have far more tokens per second than needed for general "close the blinds" "tool_call(blinds, CLOSED)".

I do wonder if more smartness could be had by using sparser experts.... And possibly even having some kind of expert switching penalty to try to reduce the amount of data read from read only flash memory by encouraging subsequent tokens to use already loaded experts.

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#165

Post by brainless »

This is awesome. I will take some time to dig in. When I am not working for client(s), I focus entirely on tiny LLMs - I have specific approach to prompting, avoid multi-turn chat and build harness to fit the selected LLM as closely as possible.

My experiments are in https://github.com/brainless/

I will be happy to share what I learn.

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#166

Post by ComputerGuru »

Can you share more about the architectural/design tradeoffs you considered or decided upon? Particularly for me, why is a model that is intended mainly to just make tool calls and marshal the results back focusing on speed? Speed as an inherent result of small size, I get, but speed as a design focus confuses me because it’s simply not going to be dealing with large outputs as a rule, wouldn’t it be better to trade some of that raw speed for better smarts?

For example, I mocked a dumbed down version of what would be a reasonable intermediate tool call prompt:

> It's currently 58 degrees. User asks for house to be 8.5 degrees warmer. What temperature to set thermostat to?

The reply?

Reasoning: “User asks for temperature to set thermostat to 8.5 -> set_thermostat with temperature=8.5.”

Sounds like something Siri would do!

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#167

Post by ComputerGuru »

ocihangir wrote: post #114
dbeardsl wrote: post #74

My first query: > Make it a little warmer in here. The reply: > "name": "set_thermostat", > "arguments": { > "temperature": 65, > "mode": "cool", > ... > "reasoning": "'warmer' implies need for cooling; set_thermostat with temperature 65 (typical warmth) and mode 'cool'.", Maybe I'm doing it wrong?

Tested your example, the confidence is 0. In smart home context, I can think of an application where the low confidence answers can be forwarded to cloud, whereas the vast majority generic queries solved locally, if the confidence is reliable enough. The response is quite fast by the way.

But this isn’t a query that should need to be forwarded to the cloud for acting on!

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#168

Post by ComputerGuru »

profsummergig wrote: post #61
HenryNdubuaku wrote: post #44

Earlier quoted context omitted.

Technically, you could do that, but we trained this one from the ground up!

That sounds like an enormously expensive exercise.

At this size, it certainly doesn’t have to be.

Re: Show HN: Needle2: 14MB agentic LLM for phones, wearables, smart home and robots

#170

Post by kube-system »

janalsncm wrote: post #64
yieldcrv wrote: post #58

what does the first L mean in LLM?

Fwiw people have told me that GPT2 doesn’t qualify as an LLM at 550MB despite being one of the first LLMs. So the practical answer to your question is: not much.

I don't think some people are aware that "large" has always referred to the training inputs not resulting the size of the model.

LLM is not "we made a language model and it is big" -- it's "we trained this model on a lot of language"

Post reply on HN