Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Explorative modeling: Train on the best of K guesses

alexiglad.github.io

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Explorative modeling: Train on the best of K guesses

#21

Post by [deleted] »

justinnk wrote: post #7
kamranjon wrote: post #4

This is amazing and I think will probably end up being a pretty important development. I was just reading this great breakdown of how diffusion Gemma works: https://newsletter.maartengrootendorst.com/p/a-visual-guide-... In reference to the difficulties with applying this to autoregressive LLMs - I wonder if these type of hybrids might be a good candidate for this approach.

I agree, this seems like an important discovery. And it hits a sweetspot of being effective and very simple to implement. Have a look at the pseudocode comparison at the bottom of the accompanying github page: https://explorative-modeling.github.io/ Also, a very well-structured and easy to follow blog post by the author. I very much enjoyed reading it!

[deleted]

Re: Explorative modeling: Train on the best of K guesses

#22

Post by zimablue22 »

yorwba wrote: post #18
zimablue22 wrote: post #14

Earlier quoted context omitted.

You're right about the original GRPO proposal, but there are simplified variants that do just use best of K sampling. GRPO (or GRPO like approaches) for diffusion/flow matching similarly can be likelihood free.

Without anchoring on the group average it's hardly Group-Relative and without likelihood there's no Policy; that doesn't leave much of the original GRPO unchanged... Do you have a particular paper in mind that does likelihood-free best-of-K but just calls it GRPO?

Any of the Best of N papers that exploded in popularity after GRPO.

Likelihood is not fundamental to the spirit of GRPO, any exploratory mechanism would work.

That sequential LLMs have a step-wise probability is convenient but not critical to this approach (where rejection sampling is widely used in diffusion models).

Re: Explorative modeling: Train on the best of K guesses

#26

Post by in-silico »

kamranjon wrote: post #24
in-silico wrote: post #19

The training method is very similar to Importance Weighted Autoencoders: https://arxiv.org/abs/1509.00519

In what way is it similar?

This algorithm: sample a bunch of latents, train the model using the one with the lowest error.

IWAE: sample a bunch of latents, weight the loss of training the model using each one by softmax(-error). For images and text where the errors have large variance, those weights become one-hot, yielding this algorithm.

Re: Explorative modeling: Train on the best of K guesses

#27

Post by kamranjon »

zimablue22 wrote: post #22
yorwba wrote: post #18

Earlier quoted context omitted.

Without anchoring on the group average it's hardly Group-Relative and without likelihood there's no Policy; that doesn't leave much of the original GRPO unchanged... Do you have a particular paper in mind that does likelihood-free best-of-K but just calls it GRPO?

Any of the Best of N papers that exploded in popularity after GRPO. Likelihood is not fundamental to the spirit of GRPO, any exploratory mechanism would work. That sequential LLMs have a step-wise probability is convenient but not critical to this approach (where rejection sampling is widely used in diffusion models).

Do you have an example? Would love to read one.

Re: Explorative modeling: Train on the best of K guesses

#28

Post by ImageXav »

This feels extremely close in nature to being a generalisation of discrete distribution networks.

Paper: https://arxiv.org/abs/2401.00036 Project page: https://discrete-distribution-networks.github.io/

Given that these were published at ICML 2025, at which the author was a top reviewer as per their own website https://alexiglad.github.io/, I wonder how influenced they were to pursue this avenue of the back of it. They do have a related works section in appendix E, but it somehow seems to miss this. Which is odd, as the paper made a splash at the time at the conference and even made it close to the top of hacker news due to the novelty.

Of course, DDNs are a fundamentally new architecture, whereas this is more a generaliseable training strategy, but there is a very similar core.

Post reply on HN