Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

github.com

Post reply on HN
Print view View original

141–150 of 255 posts

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#141

Post by jmalicki »

exabrial wrote: post #73

Dumb questions, from someone not in the field... What is a distilled model? Why doesn't Google do this (to make their models smaller)? Seems like you could make a competitor to Gemini?

There are two answers already and neither is entirely adequate.

In normal LLM training, you take a set of documents and have it learn to predict the future, then have some private RLHF/RLVR etc. data that it learns to produce good chat outputs from.

In distillation, you take a set of prompts you are interested in, and record the big LLM's outputs, then train your small model to produce the same output as the big LLM.

This has a few advantages - you can get performance much more quickly on your documents/prompts of interest, with a much cheaper training budget, and you don't have to worry about acquiring very expensive RLHF/RLVR training data.

A lot of the very good Chinese LLMs got very good very quickly through distillation from frontier models, which is why Anthropic/Google/OpenAI are blocking it so aggressively.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#142

Post by incrudible »

ai_fry_ur_brain wrote: post #137
jumploops wrote: post #132

This is neat, and matches an observation I saw with early Claude Code usage: Sonnet would often call tools quickly to gather more context, whereas Opus would spend more time reasoning and trying to solve a problem with the context it had. This led to lots of duplicated functions and slower development, though the new models (GPT-5.5 and Opus 4.6) seem to suffer from this less. My takeaway was that “dumber” (i.e. smal…

The key is to not run LLMs in loops. This trend of agentic frameworks is silly, and mostly exists to make LLM companies more revenue. An LLM is mostly useless but is much more useful and reliable with one shot tooling. I have a suite or tools ive built for myself on top of the openrouter api for very specific tasks. Press button amd LLM does (one) useful thing, not press button and let LLM run tool calls in a loop fo…

You are completely wrong, but one might get that impression from not using SOTA models in the Sonnet ballpark.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#143

Post by incrudible »

tommica wrote: post #135
madduci wrote: post #134

Earlier quoted context omitted.

Well, it's like robbing the robbers, when it comes to training data

Except one of the robberers is a massive corporation with even bigger legal team...

It is more like imitating the imitators. There is not much of a legal case here, but poisoning the data is fair game both for those producing original data as well as for those producing its regurgitations.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#145

Post by worthless-trash »

incrudible wrote: post #143
tommica wrote: post #135

Earlier quoted context omitted.

Except one of the robberers is a massive corporation with even bigger legal team...

It is more like imitating the imitators. There is not much of a legal case here, but poisoning the data is fair game both for those producing original data as well as for those producing its regurgitations.

I think its very hard for the 'websites' to poison the data for ai though, we dont have the 'single point of ingestion' to measure when its being pumped for training data.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#146

Post by NitpickLawyer »

jmalicki wrote: post #141
exabrial wrote: post #73

Dumb questions, from someone not in the field... What is a distilled model? Why doesn't Google do this (to make their models smaller)? Seems like you could make a competitor to Gemini?

There are two answers already and neither is entirely adequate. In normal LLM training, you take a set of documents and have it learn to predict the future, then have some private RLHF/RLVR etc. data that it learns to produce good chat outputs from. In distillation, you take a set of prompts you are interested in, and record the big LLM's outputs, then train your small model to produce the same output as the big LLM.…

For completeness sake I'll add a bit more.

The concept of distillation is not new in ML, and there are nuances to it. Traditionally you would have access to the bigger model, and for LLMs specifically you can train the small model on the entire distribution of output logits at the same time. So this would train the small model to output scores for each token in a similar fashion to the large model. There's "more to learn" from the entire distribution, rather than just from the chosen token.

But since you don't have access to this from the API providers, the next best thing is to use the outputs themselves and train on those. That's more like a "poor man's distillation". It's still good, and as you mentioned worked fairly well for models catching up. But a lab that develops both the big model and the small model could make it better. (or you could choose to distill from an existing open model).

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#147

Post by Balinares »

cmrdporcupine wrote: post #54
Balinares wrote: post #26

Earlier quoted context omitted.

Man, I love that there are still people writing new MOO servers in 2026. Any game out there already running on mooR?

Many people tease that they will, and start... but then kinda stop. But mostly just been building my own bespoke thing on my own bespoke platform, and kinda running out of steam because I need to make $$ instead.

Ah, sad, but not surprising. The hard part of getting a game going is assembling and sustaining a community.

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#148

Post by deivid »

brainless wrote: post #88

Lovely to see the push for tiny models. I have been building for small (20B or less) models for quite a while. Highly focused/constrained agents, many of them running together in some kind of task orchestration mode to achieve what feels like one "agent". I build (privacy first) desktop apps this way and I want to get into mobile apps with similar ideas but tiny models.

Commercial or FOSS? I've been researching the mobile side and it's very exciting!

Re: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

#150

Post by hansmayer »

jumploops wrote: post #132

This is neat, and matches an observation I saw with early Claude Code usage: Sonnet would often call tools quickly to gather more context, whereas Opus would spend more time reasoning and trying to solve a problem with the context it had. This led to lots of duplicated functions and slower development, though the new models (GPT-5.5 and Opus 4.6) seem to suffer from this less. My takeaway was that “dumber” (i.e. smal…

> and matches an observation I saw with early Claude Code

> though the new models (GPT-5.5 and Opus 4.6) seem to suffer from this less

> My takeaway was that

> haven’t found Gemini to be

For the love of all that's holy, folks please stop investing your time to fill in the gaps that the Slop Corporations are leaving wide open in their "tooling". Why should you strain yourself in an attempt to "make it work" one way or another? Google, MS, Meta, OpenAI etc. are all now subtly pushing to call their tooling "Intelligence" (not even Artificial Intelligence), so why is it not intelligent? Why does it not work? 1T+ investments and still we should think of best magic chants and configurations to make the slop generators produce half-valid output? All while some of the tech leaders are openly threatening to subdue us in their weird visions of "civilisation" ? We have a better use for our superior brains, let's not denigrate ourselves into being helpless helpers to the magic oracle (if at least it was some magic oracle!)

Post reply on HN