It's actually difficult to work out the affiliation of the authors for non-Chinese. SJTU = Shanghai Jiao Tong University, but couldn't work out GAIR and IIS.
GAIR is the generative AI lab at SJTU.
LIMO: Less Is More for Reasoning
101–110 of 137 posts
Re: LIMO: Less Is More for Reasoning
#102Here is how I make sense of it (I have no expertise in this subject, please feel free to correct me if I am wrong): I think when the model is pretrained on the internet, it does gain most of the skills required to do mathematical reasoning, however, since its task is to predict the next word distribution on the entire internet, it does not normally use this ability, since most of the text on the internet is not this…
Kinda similar to how Anthropic was able to achieve golden gate Claude or even maximize/minimize features like “buggy code” via analyzing concepts in activations and manipulating them[0]. [0]: https://www.anthropic.com/news/mapping-mind-language-model
Thank you for the link, I wasn’t aware that there were high quality blogs by Anthropic (or about golden Gate Claude).
Re: LIMO: Less Is More for Reasoning
#103I think I've recently read two seemingly contradicting things: 1- LLMs can never generalize theorem proving 2- this paper: "This suggests that contemporary LLMs may already possess rich mathematical knowledge in their parameter space, transforming the challenge from knowledge acquisition to knowledge elicitation" Not sure what is what anymore!
I think the way to swallow this bitter pill is to acknowledge they can "generalize" because all human knowledge is actually a relatively "small" finite distribution that models are now big enough to pattern match on.
And another way is that the human brain is a relatively "small" circuit that models are now big enough to model ;)
Re: LIMO: Less Is More for Reasoning
#104Earlier quoted context omitted.
LLM can never solve a halting problem (because no one can using a Turing machine).
A finite-size LLM can solve the finite-size halting problem, and an infinite-size LLM can solve the infinite-size halting problem
Halting problem input has finite size (i.e. it’s a Turing machine)
Re: LIMO: Less Is More for Reasoning
#105I'm confused.
This looks like a distillation of Qwen for math problems. What Am I missing?
Re: LIMO: Less Is More for Reasoning
#106I will believe reasoning architectures when the model knows how to store parametric information in an external memory out of the training loop.
Why is that?
Re: LIMO: Less Is More for Reasoning
#107My conclusion from all that I'm reading lately is that LLMs cannot do deduction but they can fake it real good. I mean, you wouldn't use this brand of AI to plot your path to Mars. Well, you could, BUT you'll also want to validate the path or risk dying. But this AI is good enough for Elon and his ilk. Because Elon's not gonna get into the capsule, you are. Because you are not the master of this AI, you are the valid…
Does it matter if they mimicked it better than most humans?
Re: LIMO: Less Is More for Reasoning
#108Earlier quoted context omitted.
Why is everyone is so critical of using information from a previous model to make a more efficient model. There’s nothing wrong with making progress using prior work. And increasing efficiency is progress. You wouldn’t criticize someone’s kombucha because they didn’t piece their SCOBY (symbiotic culture of bacteria and yeast) together microbe by microbe.
You are looking at it from a product perspective. From a scientific perspective, it just means the respective benchmark is meaningless, so we don't know how well such a model generalizes.
Another way to look at this is: The first assembly language compiler was handcoded in binary to begin with, and then that compiler's machine code was translated to the more expressive language (assembly). Similar for Fortran/C/etc. from assembly code. Progressively, more expressive languages have been bootstrapped from prior lower-level languages. In a similar way, perhaps a more concise LLM can be built by utilizing a less efficient one?
Re: LIMO: Less Is More for Reasoning
#109Earlier quoted context omitted.
Yes, the authors explicitly highlighted those two points in the abstract, in terms of them being the elicitation threshold for complex reasoning, namely, an extremely complete pre-trained foundation model, and a set of extremely high quality examples post-training. To your question on finetuning on the initial 10 million pool - intuitively, it would require tremendous amount of finetuning data to move the needle - yo…
The abstract doesn’t specify that the 857 training examples were filtered down by R1 from 10 million initial questions. This helps to understand the result better: it is in large part a testament to R1 and similar models’ remarkable ability sift through and identify/construct perfect training data for other models.
Isn't every progression in technology a result of the previous advance in technology enabling it?
Re: LIMO: Less Is More for Reasoning
#110Cool result, but worth highlighting two points: - Model is finetuned from Qwen-2.5 Instruct, which includes millions of specially filtered math examples in both pretraining and supervised fine-tuning already. - To generate the perfect 817 math examples for LIMO, they used state of the art models like R1 to filter down from an initial pool of 10 million math problems. In other words, a whole lot of intelligence was us…
Why is everyone is so critical of using information from a previous model to make a more efficient model. There’s nothing wrong with making progress using prior work. And increasing efficiency is progress. You wouldn’t criticize someone’s kombucha because they didn’t piece their SCOBY (symbiotic culture of bacteria and yeast) together microbe by microbe.
I'd say that the critique points out that this "information from a previous model" itself needs tremendous amounts of data. Now, did we see any better generalization capabilities with all data counted?