Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

github.com

Post reply on HN
Print view View original

111–120 of 255 posts

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#111

Post by jcgrillo »

digdugdirk wrote: post #110
jcgrillo wrote: post #106

Earlier quoted context omitted.

> people make stuff to make stuff. There don't need to be product use cases. OK. Great! So it doesn't need to be a commercial product. But does it do something (anything?) interesting ? I'm interested in your games example, I'd love to see it done in real life. IIUC, game AIs are actually much more constrained and predictable for play-ability reasons. If you let it go all free form a plurality of players have a "WTF?…

It doesn't have to do any thing interesting - it's completely fascinating all on it's own. If you understand anything about the math and science behind LLMs, you'll understand that this is an achievement worthy of sharing to a community like HN. That being said, small models like these have plenty of use cases. They allow for extra "slack" to be introduced into a programmatic workflow in a compute constrained environ…

> Something like this could help enable the "ever present" phone assistant, without scraping all your personal data and sending it off to Google/OpenAI/etc

OK so show me what that's for. Show me something useful you can do with that ability.

> Imagine if keywords in a chat would then trigger searches on your local data to bring up relevant notes/emails/documents into a cache, and then this cache directly powers your autocomplete (or just a sidebar that pops up with the most relevant information).

I'm really trying but.. idgi? I truly cannot imagine how this would improve my life in any way...

> Its cool. Enjoy it.

No. It sounds like a useless complication on my watch. I don't fucking care if it can tell me the phase of the moon. I can look up at the sky and see the moon and know what phase it is.

EDIT: You say:

> If you understand anything about the math and science behind LLMs, you'll understand that this is an achievement worthy of sharing to a community like HN.

OK. So educate me. Tell me what I'm missing.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#112

Post by michelsedgh »

nl wrote: post #99

Do you have any examples or data on the discriminatory power of the model for tool use? The examples are things like "What is the weather in San Francisco", where you are only passed a tool like tools='[{"name":"get_weather","parameters":{"location":"string"}}]', I had a thing[1] over 10 years ago that could handle this kind of problem using SPARQL and knowledge graphs. My question is how effective is it at handling…

Thanks to a Huggingface linked below, I tested it and im not impressed. prmopt: i need to contact my boss i will be late. Result: 20mins [{"name":"set_timer","arguments":{"time_human":"20 minutes"}}]. It didnt use the email tool and i tried 2-3 different ways of asking it.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#114

Post by kristopolous »

HenryNdubuaku wrote: post #29
kristopolous wrote: post #22

That M versus B is way too subtle. 0.026B is my suggestion

Haha, we were trying to not be hand-wavy too much :)

Oh hey it's Henry. I met you a couple weeks ago at an event in SF. Nice to see you on here.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#115

Post by dymk »

dang wrote: post #108
dymk wrote: post #69

Earlier quoted context omitted.

[flagged]

Can you please make your substantive points without sharp elbows? We're trying for something different here, and would appreciate it if you'd post in the intended spirit. https://news.ycombinator.com/newsguidelines.html

I’d edit it if I could, but it seems to be past the timeout.

As the other poster noted, the post wasn’t meant to be read as a personal attack

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#116

Post by HnUser12 »

michelsedgh wrote: post #112
nl wrote: post #99

Do you have any examples or data on the discriminatory power of the model for tool use? The examples are things like "What is the weather in San Francisco", where you are only passed a tool like tools='[{"name":"get_weather","parameters":{"location":"string"}}]', I had a thing[1] over 10 years ago that could handle this kind of problem using SPARQL and knowledge graphs. My question is how effective is it at handling…

Thanks to a Huggingface linked below, I tested it and im not impressed. prmopt: i need to contact my boss i will be late. Result: 20mins [{"name":"set_timer","arguments":{"time_human":"20 minutes"}}]. It didnt use the email tool and i tried 2-3 different ways of asking it.

Did you give it an email tool? It uses the tool it’s given. HF example only has timer tool.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#117

Post by mahmoudimus »

michelsedgh wrote: post #112
nl wrote: post #99

Do you have any examples or data on the discriminatory power of the model for tool use? The examples are things like "What is the weather in San Francisco", where you are only passed a tool like tools='[{"name":"get_weather","parameters":{"location":"string"}}]', I had a thing[1] over 10 years ago that could handle this kind of problem using SPARQL and knowledge graphs. My question is how effective is it at handling…

Thanks to a Huggingface linked below, I tested it and im not impressed. prmopt: i need to contact my boss i will be late. Result: 20mins [{"name":"set_timer","arguments":{"time_human":"20 minutes"}}]. It didnt use the email tool and i tried 2-3 different ways of asking it.

works for me:

input: i need to contact my boss i will be late. output: [{"name":"send_email","arguments":{"to":"boss@company.com","subject":"Running late","body":"I will be late for the meeting."}}]

it did have the send_email tool on the left hand side though

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#118

Post by Majromax »

krackers wrote: post #107
efskap wrote: post #100

No FFN is blowing my mind. This is pretty much "Attention Is ACTUALLY All You Need". Reminds me of BERT Q&A which would return indices into the input context, but even that had a FFN. Really exciting work.

I guess this had always been bugging me. I get while you need activation/non-linearities, but do you really need the FFN in Transformers? People say that without it you can't do "knowledge/fact" lookups, but you still have the Value part of the attention, and if your question is "what is the capital of france" the LLM could presumably extract out "paris" from the value vector during attention computation instead of n…

> if your question is "what is the capital of france" the LLM could presumably extract out "paris" from the value vector during attention computation instead of needing the FFN for that.

But how do you get 'Paris' into the value vector in that case? The value vector is just the result of a matrix multiplication, and without a nonlinearity it can't perform a data-dependent transformation. Attention still acts as a nonlinear mixer of previous values, but your new output is still limited to the convex combination of previous values.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#119

Post by hosh »

jcgrillo wrote: post #80
HenryNdubuaku wrote: post #63

Earlier quoted context omitted.

It is for building agentic capabilities into very small devices like phones, glasses, watches and more. Does that make sense?

[flagged]

A local model that can do better than Siri or Alexa as a personal or home assistant is, in my eyes, very useful. Being able to run on a phone or watch or glasses translates to me, low-powered AI, and not necessarily that I want my phone, or watch, or glasses to run things for me.

My Siri use has narrowed down to just setting timers. And even then, I still have my phone call people in the middle of the night. Siri is pretty dumb and does not do what I want it. I’d rather be able to customize an assistant to myself.

I am also thinking of automation in my day to day workflow for work.

Post reply on HN