Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NeurIPS 2025 Best Paper Awards

blog.neurips.cc

Post reply on HN
Print view View original

21–30 of 31 posts

Re: NeurIPS 2025 Best Paper Awards

#21

Post by Scene_Cast2 »

energy123 wrote: post #15
Scene_Cast2 wrote: post #2

I think my favorite of the bunch is the "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model" paper. Easy to read, gets the point across very intuitively and quickly, and the point is very interesting and relevant to a lot of people. About the Superposition paper - this is close to what I've been thinking about over the past week. I'm thinking that concepts or choices in a…

I am not sure how to interpret the first paper's results. If we use a random number generator then we will converge to 100% correct answers under pass@n in the limit. A random number generator will eventually outperform or match all models (for large n) whenever top-p is less than 1 because the other models will most likely have some level of bias that makes correct CoTs mathematically impossible due to the tokens be…

I'm not sure how likely it is that an answer would fall outside of the top-p of 0.95 (used in the paper). A random number generator would also need an unreasonably high number of samples to get a correct answer. I think figures 17 and 18 are interesting for this discussion too, they show performance at various sampling temperatures. I think the point of the paper is that RL "sharpens" the distribution of non-RL nets, but it does not uncover any new reasoning paths - non-RL nets already had multiple decently high probability paths of answering questions to begin with, and RL reuses a subset of those.

Re: NeurIPS 2025 Best Paper Awards

#22

Post by robrenaud »

energy123 wrote: post #15
Scene_Cast2 wrote: post #2

I think my favorite of the bunch is the "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model" paper. Easy to read, gets the point across very intuitively and quickly, and the point is very interesting and relevant to a lot of people. About the Superposition paper - this is close to what I've been thinking about over the past week. I'm thinking that concepts or choices in a…

I am not sure how to interpret the first paper's results. If we use a random number generator then we will converge to 100% correct answers under pass@n in the limit. A random number generator will eventually outperform or match all models (for large n) whenever top-p is less than 1 because the other models will most likely have some level of bias that makes correct CoTs mathematically impossible due to the tokens be…

I agree that pass@k feels a bit weird for large k. But for LLMs, it's a decent proxy for "are the knowledge/skills/circuit necessary to solve the problem somewhere in the model". Note that choices for large k is on the order of 256, and the range of valid answers is much larger than that. So your infinite monkeys critique, while true in the limit, wouldn't actually outperform models in the tested regime.

Also, in practice, models don't have that much semantic entropy of a given prompt. With temperature based sampling, models will tend to generate very similar but not identical responses.

Re: NeurIPS 2025 Best Paper Awards

#23

Post by robrenaud »

ilaksh wrote: post #3

Does some have a similar award for papers that are innovative? Like new, relatively unproven architectures?

From what I've seen at neurips, in terms of most different but maybe viable, it would be this.

https://sakana.ai/ctm/

In terms of a fresh perspective on designing learning systems, nested learning seems very interesting.

https://abehrouz.github.io/files/NL.pdf

Hearing the clarity, creativity, and force behind his thoughts and speech, I'd give a more than 1/200 chance Ali Behrouz gets himself a Turing award. At the very least, I think he will end making major contributions to AI.

Re: NeurIPS 2025 Best Paper Awards

#24

Post by boroboro4 »

energy123 wrote: post #15
Scene_Cast2 wrote: post #2

I think my favorite of the bunch is the "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model" paper. Easy to read, gets the point across very intuitively and quickly, and the point is very interesting and relevant to a lot of people. About the Superposition paper - this is close to what I've been thinking about over the past week. I'm thinking that concepts or choices in a…

I am not sure how to interpret the first paper's results. If we use a random number generator then we will converge to 100% correct answers under pass@n in the limit. A random number generator will eventually outperform or match all models (for large n) whenever top-p is less than 1 because the other models will most likely have some level of bias that makes correct CoTs mathematically impossible due to the tokens be…

To me intellect has two parts to it: "creativity" and "correctness". And from this perspective random sampler is infinitely "creative" - over (infinite) time it can come up with answer to any given problem. And from this perspective it does feel natural that base models are more "creative" (because that's what being measured in the paper), while RL models are more "correct" (that's a slope of the curve from the paper).

Re: NeurIPS 2025 Best Paper Awards

#25

Post by Der_Einzige »

cubefox wrote: post #16
FrozenSynapse wrote: post #14

Earlier quoted context omitted.

use NotebookLM

Whenever I search for the title a new machine learning paper, there are a bunch of YouTube videos about it which are just NotebookLM slop. It's straight up environmental pollution.

One of the most popular of those slop videos was about our antislop sampler. Ironic.

https://youtu.be/PHSqcdIc5gM?si=I62bduoDgnlNFPZ6

Re: NeurIPS 2025 Best Paper Awards

#27

Post by energy123 »

Scene_Cast2 wrote: post #21
energy123 wrote: post #15

Earlier quoted context omitted.

I am not sure how to interpret the first paper's results. If we use a random number generator then we will converge to 100% correct answers under pass@n in the limit. A random number generator will eventually outperform or match all models (for large n) whenever top-p is less than 1 because the other models will most likely have some level of bias that makes correct CoTs mathematically impossible due to the tokens be…

I'm not sure how likely it is that an answer would fall outside of the top-p of 0.95 (used in the paper). A random number generator would also need an unreasonably high number of samples to get a correct answer. I think figures 17 and 18 are interesting for this discussion too, they show performance at various sampling temperatures. I think the point of the paper is that RL "sharpens" the distribution of non-RL nets,…

  > I think the point of the paper is that RL "sharpens" the distribution of non-RL nets, but it does not uncover any new reasoning paths
This is an implication of the results that's intuitive and likely to be correct, but isn't guaranteed to be correct. The results do show worse answer correctness for large k. But answers and reasoning strategies to arrive at these answers are different things. It's impractical to inspect the CoTs in both the RL and Base to show that all the reasoning strategies used by the former are a subset of the latter. For all we know the venn diagram might not be fully overlapping. It could be that the RL did uncover some novel and subtle reasoning strategies not present in the Base, but it also introduced separate handicaps for some unknown reason, which nerfed answer correctness for large k. We need some theory to bridge that understanding which seems lacking in the paper? Not that I fault them for an absence of such a theory because it seems intractable. But then I am doubtful one could reach such a neat conclusion as they have tried to do, beyond the appeal to strong intuition (which I also share).

Re: NeurIPS 2025 Best Paper Awards

#28

Post by Scene_Cast2 »

energy123 wrote: post #27
Scene_Cast2 wrote: post #21

Earlier quoted context omitted.

I'm not sure how likely it is that an answer would fall outside of the top-p of 0.95 (used in the paper). A random number generator would also need an unreasonably high number of samples to get a correct answer. I think figures 17 and 18 are interesting for this discussion too, they show performance at various sampling temperatures. I think the point of the paper is that RL "sharpens" the distribution of non-RL nets,…

> I think the point of the paper is that RL "sharpens" the distribution of non-RL nets, but it does not uncover any new reasoning paths This is an implication of the results that's intuitive and likely to be correct, but isn't guaranteed to be correct. The results do show worse answer correctness for large k. But answers and reasoning strategies to arrive at these answers are different things. It's impractical to ins…

Ah, I think I agree. There could be a potential unrelated handicap, so there is a lack of a guarantee or a proof.

Re: NeurIPS 2025 Best Paper Awards

#29

Post by mountainriver »

Scene_Cast2 wrote: post #2

I think my favorite of the bunch is the "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model" paper. Easy to read, gets the point across very intuitively and quickly, and the point is very interesting and relevant to a lot of people. About the Superposition paper - this is close to what I've been thinking about over the past week. I'm thinking that concepts or choices in a…

> "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model"

I believe NVidia’s ProRL showed otherwise right?

Re: NeurIPS 2025 Best Paper Awards

#30

Post by tipsytoad »

energy123 wrote: post #15
Scene_Cast2 wrote: post #2

I think my favorite of the bunch is the "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model" paper. Easy to read, gets the point across very intuitively and quickly, and the point is very interesting and relevant to a lot of people. About the Superposition paper - this is close to what I've been thinking about over the past week. I'm thinking that concepts or choices in a…

I am not sure how to interpret the first paper's results. If we use a random number generator then we will converge to 100% correct answers under pass@n in the limit. A random number generator will eventually outperform or match all models (for large n) whenever top-p is less than 1 because the other models will most likely have some level of bias that makes correct CoTs mathematically impossible due to the tokens be…

It’s a quite deceptive paper. The main headline benchmarks (math500, aime24 /25) final answer is just a number from 0-1000, so what is the takeaway supposed to be for pass@k of 512/1024?

On the unstructured outputs, where you can’t just ratchet up the pass@k until it’s almost random, it switches the base model out for instruct, and in the worse case on livecodebench it uses a qwen-r1-distill as a _base_ model(!?) that’s an instruct model further fine tuned on R1’s reasoning traces. I assume that was because no matter how high the pass@k, a base model won’t output correct python.

Post reply on HN