Can LLMs learn from a single example? • Hacker News Board

Re: Can LLMs learn from a single example?

#141

Post by t-vi » Thu, Sep 07, 2023, 5:57 PM UTC

famouswaffles wrote: post #3

jph00 wrote: post #2

Thank you for posting this to HN! :D I'm one of the authors of this post -- Johno & I found it really interesting looking into this curious issue of rapid memorization from LLMs. I've been working with neural nets for 30 years, and fine-tuning language models since 2017, and this behavior is most surprising to me! Other folks have seen it in LLMs too, although I haven't seen a analysis of this kind before (although w…

In the palm-e paper ( https://palm-e.github.io/ ), when they try to unfreeze and train the LLM on new image data only, there is expectedly a lot of CF on NLP tasks but very interestingly, the effect diminishes greatly with the scale of the LLM prior to training. From an average -87.3% performance drop on the 12B model to -61.6% on the 84B model then just -3.9% on the 562B model. Felt like we were just shy of an insig…

> Is avoiding CF potentially just a matter of sheer scale ?

My intuition would be that you get more orthogonal directions to the gradient (of previous samples) if you have larger model.

Re: Can LLMs learn from a single example?

#142

Post by armatav » Thu, Sep 07, 2023, 11:57 PM UTC

niemandhier wrote: post #138

armatav wrote: post #130

Earlier quoted context omitted.

Is that necessarily the case for massive LLMs, or could there be a more refined grouping?

I am not sure if the origin of polysyntacicity is fully understood. From a physics perspective it’s entropy: There are just more local minima that have neurons code multiple things. I suspect that dropout and similar tricks also play a role in this: Removing connections during training means that pathways need to be redundant somewhat.

That's super interesting, I wonder if dropout applied in a certain pattern of segregation could induce more neuron dependence, instead of less. Or do some anti-dropout where a dedicated portion of the model is used. Not sure why you'd want that, but it might be interesting to explore.

Re: Can LLMs learn from a single example?

#143

Post by kordlessagain » Sat, Sep 09, 2023, 5:11 PM UTC

tudorw wrote: post #137

kordlessagain wrote: post #113

Earlier quoted context omitted.

I have code stubbed out for this in mitta.us. It has 9 states, based on the Plutchik wheel, with emojis for the states. States drive temp and a few other things and drop the state into prompts.

Interesting, do you have a mailing list or way I can be notified of progress?

You can signup: https://mitta.us/

The accounts aren't wired up by default to the AI and I am refactoring the templating system right now, but you can definitely start storing and searching things.