Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Explorative modeling: Train on the best of K guesses

alexiglad.github.io

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Explorative modeling: Train on the best of K guesses

#31

Post by angusturner »

I find the presentation of this as "factoring training" very confusing.

Factoring is something we do to distributions, which we then parameterize with our neural net.

An autoregressive model pertains to a certain factorisation of a joint distribution. And if we choose this factorisation it may have consequences for our training and sampling etc.

But to say "most models on factor generation".. I dont quite get this.

This would be better work if it didnt claim it was a new pretraining axis also. Its another way of scaling compute right?

If it were a new axis, IWAE would have an equal claim to it (as others have pointed out).

Re: Explorative modeling: Train on the best of K guesses

#32

Post by SwellJoe »

Every time I read something like this, I'm thinking, "The model knows where it is at all times. It knows this because it knows where it isn't. By subtracting where it is from where it isn't, or where it isn't from where it is (whichever is greater), it obtains a difference, or deviation." At least the first couple of times through it.

Re: Explorative modeling: Train on the best of K guesses

#34

Post by 56767865678 »

cousin_it wrote: post #5

No mention of GANs? As far as I remember, their images felt much more like "sampled from training data" than "averaged from training data". The idea was that you train a "discriminator" that tells generated images apart from real ones, and make the "generator" try to fool it. But somehow it lost to diffusion models, and now all AI imagery looks like slop, when GAN outputs (though imperfect) didn't look nearly as slop…

[flagged]

Re: Explorative modeling: Train on the best of K guesses

#35

Post by woadwarrior01 »

justinnk wrote: post #7
kamranjon wrote: post #4

This is amazing and I think will probably end up being a pretty important development. I was just reading this great breakdown of how diffusion Gemma works: https://newsletter.maartengrootendorst.com/p/a-visual-guide-... In reference to the difficulties with applying this to autoregressive LLMs - I wonder if these type of hybrids might be a good candidate for this approach.

I agree, this seems like an important discovery. And it hits a sweetspot of being effective and very simple to implement. Have a look at the pseudocode comparison at the bottom of the accompanying github page: https://explorative-modeling.github.io/ Also, a very well-structured and easy to follow blog post by the author. I very much enjoyed reading it!

At first glance, this work might also be relevant for quantizing diffusion transformers (something I'm working on at the moment).

Re: Explorative modeling: Train on the best of K guesses

#36

Post by FeepingCreature »

cousin_it wrote: post #5

No mention of GANs? As far as I remember, their images felt much more like "sampled from training data" than "averaged from training data". The idea was that you train a "discriminator" that tells generated images apart from real ones, and make the "generator" try to fool it. But somehow it lost to diffusion models, and now all AI imagery looks like slop, when GAN outputs (though imperfect) didn't look nearly as slop…

Iirc it lost to diffusion models because it was extremely hard to train. Due to the two adversarial networks it was very prone to mode collapse, ie. the generator running ahead of the discriminator and destroying the training signal. Or other various fun ways the two networks could blow each other up.
Post reply on HN