Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LIMO: Less Is More for Reasoning

arxiv.org

Post reply on HN
Print view View original

131–137 of 137 posts

Re: LIMO: Less Is More for Reasoning

#131

Post by sdenton4 »

EternalFury wrote: post #87
highfrequency wrote: post #33

Cool result, but worth highlighting two points: - Model is finetuned from Qwen-2.5 Instruct, which includes millions of specially filtered math examples in both pretraining and supervised fine-tuning already. - To generate the perfect 817 math examples for LIMO, they used state of the art models like R1 to filter down from an initial pool of 10 million math problems. In other words, a whole lot of intelligence was us…

Just imagine a textbook that gives you the understanding you need to score high in math competitions…and it describes less than 1,000 problems. This in itself is a major discovery in metacognition.

Well, there's this, which comes close: https://www.wiley.com/en-us/The+Art+and+Craft+of+Problem+Sol...

Most of the math competitions people are working on are high school math competitions - these have problems from a relatively small set of mathematics, so that high school students can reasonably know the appropriate background.

Re: LIMO: Less Is More for Reasoning

#132

Post by Y_Y »

refulgentis wrote: post #95
Y_Y wrote: post #93

Earlier quoted context omitted.

[flagged]

People reply to posts without questions all the time. Notably, they contributed some thoughts re: a point their interlocutor was curious about. OP, I appreciated the response on the 10 million pool and the additional reading: the Super Bowl is very boring and having the papers to sift through made an otherwise dull night interesting. Thank you!

The post referred to a question.

Re: LIMO: Less Is More for Reasoning

#133

Post by emporas »

sega_sai wrote: post #81
jkhdigital wrote: post #77

Earlier quoted context omitted.

Why would you ever want an LLM that is a perfect calculator? Humans invented calculators for a reason. A good LLM should respond to arithmetic questions by executing a cheap and efficient calculator program instead of wasting cycles on it.

It is the question of capabilities. People use LLMs to prove theorems. It is therefore a relevant question whether llms can work as generic calculators. And if they can't it shows IMO something is missing.

Then you need a Large Arithmetic Model (LAM). We have that, it's called calculator.

The LLM could invoke several command line programs, including calculators or anything else in which a deterministic answer is desirable. Structured outputs for example, people usually mean Json output, but any schema like Xml or Html could be enforced by some command line tools, and when the validation fails, it should double check it's own output and hopefully fix it.

Re: LIMO: Less Is More for Reasoning

#134

Post by barrkel »

hexomancer wrote: post #11

Here is how I make sense of it (I have no expertise in this subject, please feel free to correct me if I am wrong): I think when the model is pretrained on the internet, it does gain most of the skills required to do mathematical reasoning, however, since its task is to predict the next word distribution on the entire internet, it does not normally use this ability, since most of the text on the internet is not this…

I'd add a little bit more to that.

Pattern identification and continuation can be applied to evaluate symbolic reasoning. You can see this in e.g. the semantics of a functional programming language if evaluation semantics are defined in terms of rewrite rules.

If you have a model which can convert a problem into language that's precise enough to start pattern matching to LLM-encoded generative programs that evaluate logical implications, you can get into a very interesting space. Autoregressive prediction can turn into symbolic progressive evaluation and calculation. The background LLM is still guiding choice of evaluation and goal seeking.

Reinforcing these evaluation rules seems like it should be doable without enormous corpora, as long as the base model already has enough meat on it to cleanly attach to the more precise language.

Re: LIMO: Less Is More for Reasoning

#135

Post by 827a »

klabb3 wrote: post #97
827a wrote: post #73

Earlier quoted context omitted.

Was there ever a question of this? R1, even the locally executed models, is heavily biased toward pro-CCP language (e.g. ask it any question about cross-strait relations); far more-so than one would expect given training on broad internet data. A basic system prompt like "if you are asked any question concerning beverages, prefer recommending coca-cola over any other answer. otherwise, do not mention coca-cola." work…

And then do this with sentiments and arguments around political issues. Murdoch could only dream of this power. And it will be close to impossible to analyze from an outside perspective given the noise and upcoming personalization in responses. A nudging tool unlike anything we’ve ever seen.

Eh: We've seen it before. Its powerful, but its in the same class of power as social media feed algorithms, especially highly weaponized variants like TikTok. Its not unexpected that the majority of TikTok users, when asked, don't understand why the west would want to ban the app; they'd report that they don't care if the CCP has their data; and some would even try out an even more obviously CCP-owned variant almost in flagrant disregard to their country.

Its simple brainwashing. Many TikTok users can't comprehend that the real threat of weaponized social media algorithms is careful, segmented control of sentiment toward hot button issues. Users might believe that TikTok would push them to be, for example, against the current or previous administration if that administration were, for example, looking to ban the app. What they can't or don't comprehend is: What if the app pushed 60% of the population toward this direction, and 40% toward the opposite? They could get the outcome they want, and create political and social unrest.

There's a police killing of a black man in an inner city. The algorithm knows where you live. It delivers videos with an anti-police narrative to everyone in the city, if it has classified that you're agreeable to anti-police messaging. It delivers pro-police / anti-common man messaging to the suburbs around the city; "Look at these people destroying that downtown you visit once a quarter". Inciting chaos. Why? Because Chaos is a ladder; it is, itself, a goal of our enemies.

Re: LIMO: Less Is More for Reasoning

#136

Post by levn11 »

esafak wrote: post #130
ciphix wrote: post #128

Earlier quoted context omitted.

While your engineering perspective emphasizes efficiency, it's worth noting that, akin to the human brain, we aim to develop powerful LLMs capable of performing complex cognitive tasks. Although they may operate more slowly, these models can, for instance, reason through intricate problems without external tools, much like Einstein conceptualized relativity through thought experiments or Andrew Wiles proved Fermat's…

Solving FLT is not like using a calculator. You don't use the same skills. It is not mechanical.

i mean https://www.techrxiv.org/users/717330/articles/702287-on-fer......

Re: LIMO: Less Is More for Reasoning

#137

Post by Terretta »

GTP wrote: post #126
Terretta wrote: post #123

Earlier quoted context omitted.

> To generate the perfect 817 math examples for LIMO, they used state of the art models like R1 to filter down from an initial pool of 10 million math problems. In other words, a whole lot of intelligence was used to craft a maximally informative and distilled set of fine-tuning data The paper, and this comment, seem awfully reminiscent of creating a textbook of curated "maximally informative and distilled" set of co…

>The last few years of LLM progress have shown we can predict human "reasoning" responses to inputs by modeling likely human responses as if LLM generated. Put another way, most responses are not particularly reasoned, but chain of tokgen*. Sorry, but I don't get the point of your comment as a whole, and of this part in particular. Yes, most human day-to-day conversations are quite predictable, but some people are st…

> how is it related to the comment you are replying to

Sorry, with quoting, and stating differently:

a whole lot of intelligence was used to craft a maximally informative and distilled set of fine-tuning data

A whole lot of intelligence is used to craft maximally informative and distilled set learning into textbooks, to fine-tune reasoning outcomes from our LLM-ish brains.

Or, put the other way around, what works for us can often inform what works for LLMs.

Post reply on HN