Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

arxiv.org

Post reply on HN
Print view View original

21–30 of 226 posts

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#21

Post by nsingh2 »

SwellJoe wrote: post #17

It's terrible at hunting security bugs (I expected it to be, but I wanted to be sure). I added it to a benchmark I made with a corpus of some Mythos-discovered bugs, and it found zero. The smallest pretty successful models remain Qwen 3.6 and Gemma 4 (but I haven't tested the very small variants of those yet). https://swelljoe.com/post/will-it-mythos/

The lack of tool use will hinder it a lot I think, since bug hunting requires collecting context across a code base and stitching it together. It might be good in a more narrow sense, i.e "is there a bug in this block of code" and not considering how it interacts with the rest of the code base.

That's also more aligned to its leetcode style training data, the code under test is fully in the context window. It might be interesting to have a bigger tool use model go through the effort of collecting the context, and feeding it into this kind of model for analysis only. It becomes more of a thinking tool, instead of the orchestrator.

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#22

Post by Mtinie »

steve_adams_86 wrote: post #18
pylotlight wrote: post #13

Earlier quoted context omitted.

SVG generation is a useless test, what's there more to know?

What if you're reasoning about how to generate SVG correctly?

In this case, I’d expect it should make a web search tool call to find the Python library best suited for SVG generation and manipulation, and then use what it learns there to execute the task you’ve asked it to do (either asking if you’d like to incorporate the library as a dependency or to roll its own implementation of a subset of the features if that was your preference),

Assuming tool calling hasn’t been entirely stripped out of this model.

(Edit) No tool calling, per this comment: https://news.ycombinator.com/item?id=48640189

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#23

Post by secretslol »

Am I right in thinking this is a tiny model which has been trained well to reason, and that's it? Makes me think of a smart person who doesn't know anything about a given topic, but with the right tools will go and research the heck out of it. I really like the sound of this... why have models train on learning anything when you can just train them how to learn and let them get on with it from something as small as a Pi Zero and an internet connection.

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#25

Post by nsingh2 »

gslepak wrote: post #14

Note that these are Python-only results, the model will not do as well with other languages. I'm glad to see more domain-focused SLMs, we need more of them! A programming focused MoE should work well across many languages.

Lots of confusion about what this model is actually focused on.

It is a cheap specialist for closed-world, verifiable reasoning tasks like math, self-contained coding problems, and similar.

"Closed-world" means the needed information is already in the context. It is not a tool-using agent that can discover missing context. "Verifiable" means answers are hard to generate but easy to check.

So no open ended research, repo wide agent work, factual Q&A, or SVG generation. More of a compact reasoning module for bounded problems.

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#26

Post by numlock86 »

secretslol wrote: post #23

Am I right in thinking this is a tiny model which has been trained well to reason, and that's it? Makes me think of a smart person who doesn't know anything about a given topic, but with the right tools will go and research the heck out of it. I really like the sound of this... why have models train on learning anything when you can just train them how to learn and let them get on with it from something as small as a…

This has been my dream ever since. Instead of encoding "all the knowledge" into those parameters, how about just making a model that has the same size, but all (or rather most) it does is reasoning? Just give it the ability to browse the net (e.g. language specifications, documentation and best practices) and just have it do its thing. Why does my coding agent need to know the population of New York, know a cheese cake recipe or the general lifespan of an ostrich? Just give it the bare minimum knowledge to think and reason about, and let it figure out the rest.

Sadly that's not how LLMs work, since all they do is "token prediction". At least the models we have to today ...

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#27

Post by 3eb7988a1663 »

numlock86 wrote: post #26
secretslol wrote: post #23

Am I right in thinking this is a tiny model which has been trained well to reason, and that's it? Makes me think of a smart person who doesn't know anything about a given topic, but with the right tools will go and research the heck out of it. I really like the sound of this... why have models train on learning anything when you can just train them how to learn and let them get on with it from something as small as a…

This has been my dream ever since. Instead of encoding "all the knowledge" into those parameters, how about just making a model that has the same size, but all (or rather most) it does is reasoning? Just give it the ability to browse the net (e.g. language specifications, documentation and best practices) and just have it do its thing. Why does my coding agent need to know the population of New York, know a cheese ca…

It would also reduce training costs to nothing. Current methodology requires continual retraining to scoop up new facts. If you can do a one time "this is how to think" - that could conceptually work forever, just plug in a new database layer that can be queried as required.

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#28

Post by smokel »

deftio wrote: post #16

There is some base level of intelligence any model needs to be useful, even in narrow tasks. Could you teach a 5 year old to drive a car? A 10 year old? A 12 year old? To drive a car requires being able to read, to have judgement about ice or rainy conditions, to anticipate a child running after a ball. By the time a human in in their mid teens they have acquired the base knowledge... Small models need to have enough…

Being able to drive a car properly also depends on having the right exploration-exploitation balance. A three-year-old is likely to explore too much in a situation where mistakes can be dangerous.

This requires not only knowledge, but also the control systems that develop with the prefrontal cortex. LLMs don't do much control yet.

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#29

Post by NotSuspicious »

The interesting thing about models this small is they should be able to be put on a single Taalas chip (the HC1 already runs a Llama 3.1 8B model). We're already at the point where half-decent reasoning could be run on an ASIC (and at mind-boggling speeds).

Re: VibeThinker: 3B param model that beats Opus 4.5 on reasoning with novel SFT+GRPO

#30

Post by ygjb »

deftio wrote: post #16

There is some base level of intelligence any model needs to be useful, even in narrow tasks. Could you teach a 5 year old to drive a car? A 10 year old? A 12 year old? To drive a car requires being able to read, to have judgement about ice or rainy conditions, to anticipate a child running after a ball. By the time a human in in their mid teens they have acquired the base knowledge... Small models need to have enough…

> Could you teach a 5 year old to drive a car? A 10 year old? A 12 year old? To drive a car requires being able to read, to have judgement about ice or rainy conditions, to anticipate a child running after a ball. By the time a human in in their mid teens they have acquired the base knowledge...

I would be interested to see a formal study of this. I say this not out of anything other than a observation that I think the only real blockers are a) judgement, and b) physical reflexes/strength. As a kid I was certainly aware of ice,snow, and rain, because I road my bike year round and had low confidence in my own ability to control my bike on snowy or wet terrain, especially during season changes. That translated into learning to drive in northern Canada in the winter and applying those lessons to driving.

In an environment devoid of consequences, I have seen kids operate driving simulations (both real simulations, and video games) with a degree of precision that is shocking, including seeing several 9-11 year olds play the simulations and games with a much higher degree of confidence than adult drivers. Children have an awareness that the simulations are consequence free, unless given other motivation. Adults that are consistent drivers have muscle memory and preconceived expectations that govern the decisions they make when playing the game. I am curious about the level of training and exposure required for children to overcome their lack of awareness of the hard limits and consequences of driving and driver error, versus the amount of training and exposure required for expert drivers that are novice gamers to stop applying their learned experience to consequence free simulations.

Post reply on HN