Grokking
domluna.com
Grokking
1–10 of 31 posts
Re: Grokking
#2If you are interested in grokking you should have a look at this: Progress measures for grokking via mechanistic interpretability. https://arxiv.org/abs/2301.05217
Re: Grokking
#3[deleted]
Re: Grokking
#4If you are interested in grokking you should have a look at this: Progress measures for grokking via mechanistic interpretability. https://arxiv.org/abs/2301.05217
https://en.wikipedia.org/wiki/Gestalt_psychology check dis too yo
"Gestalt psychologists emphasize that organisms perceive entire patterns or configurations, not merely individual components."
Re: Grokking
#5Did this originate in "stranger in a strange land"?
I assumed it did, but so many references to grokking don't mention the book (which goes into great detail about what it means)
Re: Grokking
#6Did this originate in "stranger in a strange land"? I assumed it did, but so many references to grokking don't mention the book (which goes into great detail about what it means)
it does.
Re: Grokking
#7Did this originate in "stranger in a strange land"? I assumed it did, but so many references to grokking don't mention the book (which goes into great detail about what it means)
Where else would it come from? It’s just using the normal meaning of “to grok” [0] (or its present participle), which is a well-known term of hacker jargon [1].
Re: Grokking
#8There was a post about v shapes in learning, both human and machine on the front page of HN a few days ago (can’t find the link).
Is this the same thing?
Re: Grokking
#9I hate this word with the burning passion of a thousand suns.
Re: Grokking
#10There was a post about v shapes in learning, both human and machine on the front page of HN a few days ago (can’t find the link). Is this the same thing?
Parent comment refers Double Descent in Human Learning [1, 2] which references the Deep Double Descent phenomenon in Large models [3].
The gist of double descent is that there exists a phenomenon where a _large_ model appears to over-fit as we expected from traditional ML, but then the val loss starts decreasing. The general consensus is that the model switches from memorization mode to interpolation mode which enables generalization.
In essence, the large number of parameters initially fit directly to the data, but then smooth out.
The reason we can continue improving the model is the fact that over-parameterized models always have a descent direction, simply due to the dimensionality of the model.
The paper that the post references, shows that by trading parameters with more training steps, we can have small models that exhibit similar if not identical behaviour.
[1] https://news.ycombinator.com/item?id=35683754
[2] https://chris-said.io/2023/04/21/double-descent-in-human-lea...