Live data from Hacker News

Learning Theory from First Principles [pdf]

di.ens.fr

21–30 of 66 posts

Re: Learning Theory from First Principles [pdf]

#21
This is pretty hard to read. For example, on the first page of chapter 1, it talks about "minimization of quadratic forms" and shows what looks like the formula for linear least squares. Is that right? It doesn't say anything about this. Some more exposition would help.

I do like that there are lots of exercises.

Re: Learning Theory from First Principles [pdf]

#22
post #14

Earlier quoted context omitted.

Ypur idea would become even better so if you read the book yourself ... You might even learn something new along the way.

we are optimizing for time here, not learning. I'm gonna die anyways and anything I learn will be dust. If I just need the info to make something work, spending months (of which I have limited number of) to see if something has something useful in it for me vs. spending and afternoon to probe it to get most of the benefits is a no-brainer.

Maybe, but dont expect otherwise from a book explicitly named "Learning Theory from First principles", not "learn large language models in 21 days".

Re: Learning Theory from First Principles [pdf]

#23

This is pretty hard to read. For example, on the first page of chapter 1, it talks about "minimization of quadratic forms" and shows what looks like the formula for linear least squares. Is that right? It doesn't say anything about this. Some more exposition would help. I do like that there are lots of exercises.

I think the text is geared towards people with some mathematical background who want to understand learning theory. Besides it is clearly stated that this chapter is a review (so its assumed that you learned or will learn these things elsewhere).

Re: Learning Theory from First Principles [pdf]

#24
post #9
post #3

I can’t wait until I tell GPT-5 “I have this idea I want to try, read this book and tell me there’s anything relevant there to make it work better”.

I've never heard of an LLM making up a new idea. Shouldn't this only work if your thing has already been tried before?

How people define "new" varies a lot in this context. I've spent a lot of time talking with ChatGPT exploring interdisciplinary ideas and while I think it frequently says things that qualify as new, I've only run into one situation where I was trying to find some way of doing something technical and it just invented something non-trivially original to handle it: https://x.com/Westoncb/status/1763733064478335326?s=20

Re: Learning Theory from First Principles [pdf]

#25

This is pretty hard to read. For example, on the first page of chapter 1, it talks about "minimization of quadratic forms" and shows what looks like the formula for linear least squares. Is that right? It doesn't say anything about this. Some more exposition would help. I do like that there are lots of exercises.

The sibling comment is right in that this is clearly not intended for first timers.

But your instincts are correct here. When you write out the objective function for ordinary least squares, it turns out to be a quadratic form. The choice of the word "quadratic" here is not a coincidence: it is the generalization of quadratic functions to matrices. That section covers the vector equivalent of minimizing quadratic functions.

Re: Learning Theory from First Principles [pdf]

#26

This is pretty hard to read. For example, on the first page of chapter 1, it talks about "minimization of quadratic forms" and shows what looks like the formula for linear least squares. Is that right? It doesn't say anything about this. Some more exposition would help. I do like that there are lots of exercises.

I think the text is geared towards people with some mathematical background who want to understand learning theory. Besides it is clearly stated that this chapter is a review (so its assumed that you learned or will learn these things elsewhere).

Well I have some math background but that section is brisk and slow at the same time, as it were. Such as how it explains how to find inverses of 2x2 matrices.

This is older but is supposed to be good: https://www.deeplearningbook.org/

Re: Learning Theory from First Principles [pdf]

#27

This is pretty hard to read. For example, on the first page of chapter 1, it talks about "minimization of quadratic forms" and shows what looks like the formula for linear least squares. Is that right? It doesn't say anything about this. Some more exposition would help. I do like that there are lots of exercises.

Certainly doesn't seem like first principles...

Re: Learning Theory from First Principles [pdf]

#28

Interesting! I’ll have to look over it when I have more time. From a quick glance, it looks like it covers much of the same material as this text [1]. I wonder how they compare. [1]: https://www.cambridge.org/core/books/understanding-machine-l...

A 2014 book on machine learning sounds quaint and historical.

It depends on the particular subtopics it covers. Machine Learning: A Probabilistic Perspective is from 2012 and it's still a great resource, although Murphy's newer book will certainly cover more up-to-date material.

Re: Learning Theory from First Principles [pdf]

#29
post #6

There are so many great mathematical PDFs available for free on the Internet, written by academics/educators/engineers. A problem is that there is a huge amount of overlap. I wonder if an AI model could be developed that would do a really good job of synthesizing an overlapping collection into a coherent single PDF without duplication.

No need for an AI model. Probabilistic Machine Learning by Murphy is an excellent reference and resource.

Re: Learning Theory from First Principles [pdf]

#30

This is pretty hard to read. For example, on the first page of chapter 1, it talks about "minimization of quadratic forms" and shows what looks like the formula for linear least squares. Is that right? It doesn't say anything about this. Some more exposition would help. I do like that there are lots of exercises.

Least squares is quadratic.

Quadratic means square terms.

Post reply on HN