Learning Theory from First Principles [pdf]
1–10 of 66 posts
Re: Learning Theory from First Principles [pdf]
#2Interesting! I’ll have to look over it when I have more time.
From a quick glance, it looks like it covers much of the same material as this text [1]. I wonder how they compare.
[1]: https://www.cambridge.org/core/books/understanding-machine-l...
Re: Learning Theory from First Principles [pdf]
#3I can’t wait until I tell GPT-5 “I have this idea I want to try, read this book and tell me there’s anything relevant there to make it work better”.
Re: Learning Theory from First Principles [pdf]
#4Have they figured out what causes double descent yet?
Re: Learning Theory from First Principles [pdf]
#5Have they figured out what causes double descent yet?
No. We don't know. My favorite hypothesis: SGD is...well, stochastic. Meaning you're not optimizing w.r.t the training corpus, but a tiny subset, so your gradient isn't quite right. Over-training allows you to bulldoze over local optima and recurse toward the true distribution rather than drive around a local over-fitting basin.
Re: Learning Theory from First Principles [pdf]
#6There are so many great mathematical PDFs available for free on the Internet, written by academics/educators/engineers. A problem is that there is a huge amount of overlap. I wonder if an AI model could be developed that would do a really good job of synthesizing an overlapping collection into a coherent single PDF without duplication.
Re: Learning Theory from First Principles [pdf]
#7Have they figured out what causes double descent yet?
No. We don't know. My favorite hypothesis: SGD is...well, stochastic. Meaning you're not optimizing w.r.t the training corpus, but a tiny subset, so your gradient isn't quite right. Over-training allows you to bulldoze over local optima and recurse toward the true distribution rather than drive around a local over-fitting basin.
You can get it with full gradient descent though... https://www.nature.com/articles/s41467-020-14663-9
Honestly the fact that there doesn't seem to be a good explanation for this makes me think that we just fundamentally don't understand learning.
Re: Learning Theory from First Principles [pdf]
#8I can’t wait until I tell GPT-5 “I have this idea I want to try, read this book and tell me there’s anything relevant there to make it work better”.
[deleted]
Re: Learning Theory from First Principles [pdf]
#9I can’t wait until I tell GPT-5 “I have this idea I want to try, read this book and tell me there’s anything relevant there to make it work better”.
I've never heard of an LLM making up a new idea. Shouldn't this only work if your thing has already been tried before?
Re: Learning Theory from First Principles [pdf]
#10Have they figured out what causes double descent yet?
There was actually a very recent blog post claiming that statistical mechanics can explain double descent https://calculatedcontent.com/2024/03/01/describing-double-d...
Some more detail here: https://calculatedcontent.com/2019/12/03/towards-a-new-theor...