Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

arxiv.org

Post reply on HN
Print view View original

131–140 of 177 posts

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#131

Post by tnecniv »

chongli wrote: post #123
up2isomorphism wrote: post #105

[flagged]

People tend to hate things they don't understand more than the things that are wrong. The comments under reflect to some degree about this observation. I have noticed that most HN discussions on math topics seem to devolve into complaints about notation. There seems to be a fairly large contingent of people from the programming side of things who don't have a formal background in mathematics. I get the feeling that i…

Not just understanding, but how to parse it. For example, I can read a theorem that introduces a bunch of variables in the first sentence that don’t get used until a few lines later and say “yeah those are probably just some constants they’ll use in an upcoming equation for an upper bound or something.” Before I was familiar with higher math, I’d say “what the hell are these where did they come from?” After reading enough books and papers, you can see a few steps ahead a lot of the time due to past experience (similar to what veteran chess players do). I can also jump to the middle of the document and see some notation that was defined earlier and guess what it is. Different authors might denote the space of continuous functions in a few different ways but they tend to be very similar for common objects.

That said, there’s a lot of poorly written theory publications out there in my opinion. A big sin to me is that a lot of them will do stuff without explaining in simple English beforehand how they’re going to do it, why they’re doing it, and why it is important. The first item you can get if you’re going through the math line by line at least, but that can be arduous and often I want to understand the big picture before I dig into the details. It doesn’t take much — just a couple of sentences before or after the result can go a long way.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#132

Post by tnecniv »

two_in_one wrote: post #110

It's hard to call comprehensive. Transformers - one page. A picture would be nice. No "prompt engineering", no "double deep". In fact words "prompt" and "double" aren't used at all. "Recognition" is used only once outside of bibliography just for reference. Looks like theory will not catch up with practice any time soon. With looming singularity it's bit worrying.

Right in the title it explains it’s a book on theory. “Prompt engineering” doesn’t really fit in any theoretical framework I’m aware of and, while I also like graphics, most theory publications are light on them. You might be looking for a different kind of book, which is fine, but I think the content matches the title.

Also, a book on theory is going to lag quite a bit in terms of topics. The general process is that people discover something new and interesting empirically and publish articles on it. Other people develop theory explaining why that things work and publish articles on it. Once the theory gets crystallized, big ideas get distilled into a book.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#133

Post by zwaps »

angra_mainyu wrote: post #66
youainti wrote: post #55

Earlier quoted context omitted.

As a PhD econ student, the mathematics just comes down solving constrained optimization problems. Figuring out what to consider as an optimand and the associated constraints is the real kicker.

If you're referring to micro/macro, I meant more like a mathematical introduction to the models. I recall giving Mankiw a try and wished I could just find a physics-style textbook as I found it way too wordy.

Most economists (who write these sort of textbooks) have some sort of math background. The push to find the most general "math" setting has been an ongoing topic since the 50's and so you can probably find what you are looking for. It's not part of undergraduate textbooks since adding generality gives better proofs but often adds "not that much" to insight. Nevertheless, the standard micro/macro models are just applications of optimization theory (lattice theory typically for micro, dynamical systems for macro). Game theory (especially mechanism design) is a bit of different topic, but I suppose that's not what you are looking for.

E.g., micro models are just constrained optimization based on the idea of representing preference relations over abstract sets with continuous functions. So obviously, the math is then very simple. This is considered a feature. You can also use more complex math, which helps with certain proofs (especially existence and representation).

You could grab some higher level math for econ textbooks, which typically include the models as examples, where you skip over the math.

For example, for micro, you can get the following: https://press.princeton.edu/books/hardcover/9780691118673/an... I think it treats the typical micro model (up to oligopoly models) via the first 50 or so pages while explaining set theory, lattices, monotone comparative statics with Tarski/Topkis etc.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#134

Post by chongli »

tnecniv wrote: post #131
chongli wrote: post #123

Earlier quoted context omitted.

People tend to hate things they don't understand more than the things that are wrong. The comments under reflect to some degree about this observation. I have noticed that most HN discussions on math topics seem to devolve into complaints about notation. There seems to be a fairly large contingent of people from the programming side of things who don't have a formal background in mathematics. I get the feeling that i…

Not just understanding, but how to parse it. For example, I can read a theorem that introduces a bunch of variables in the first sentence that don’t get used until a few lines later and say “yeah those are probably just some constants they’ll use in an upcoming equation for an upper bound or something.” Before I was familiar with higher math, I’d say “what the hell are these where did they come from?” After reading e…

A big sin to me is that a lot of them will do stuff without explaining in simple English beforehand how they’re going to do it, why they’re doing it, and why it is important.

I don't fault theoreticians for not using simple English. In many cases you're dealing with objects that are built upon a tower of abstractions with which you and your colleagues are already intimately familiar. This is true in any technical field. Sit in a hospital cafeteria long enough and I'm sure you can overhear surgeons talking shop over lunch. I wouldn't expect them to use simple English either. They have a huge corpus of terminology for every muscle, tendon, ligament, and bone in the body. Skipping past the simple English allows them to be brief and fluid in their communication style, at the expense of leaving laypeople out of the loop.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#135

Post by aurareturn »

tnecniv wrote: post #61
HybridCurve wrote: post #25

As someone who has a deeper knowledge of programming rather than math, I find the mathematical notation here to be harder to understand than the code (even in a programming language I do not know). Does anyone with a stronger mathematical background here find it easier to understand the math as written more easily than the source code?

So this is a book written by applied mathematicians for applied mathematics (they state in the preface it’s for scientists, but some theoretical scientists and engineers are essentially applied mathematics). As a result, both the topics and the presentation are biased towards those types of people. For example, I’ve never seen in practice worry about the existence and uniqueness conditions for their gradient-based op…

Are there any books you recommend for deep learning that are written for developers who don't use math every day?

I suppose the goal would be to understand deep learning so that we know enough of what's going on but not to get stuck in math concepts that we probably don't know and won't use.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#136

Post by modeless »

c7b wrote: post #70

Seems like a good collection of standard ML techniques, introduced with a fairly unified mathematical notation and quite a few proofs. Quite the Herculean effort (600 pages!). It just seems to me like they're putting the emphasis on the stuff that is more straightforward to formalize rather than the stuff that would be interesting to understand. Look eg at the SGD chapter. I picked this because I think optimization i…

> they won't necessarily be much wiser about why those methods work, other than having a good intuition confirmed by numerical experiments.

This is the state of the field as a whole, isn't it?

> Wouldn't it be more useful to dedicate the space to convergence proofs for ADAM (which do exist)

Convergence proofs don't really explain why Adam tends to work better than other methods.

It's hard to blame them for not being able to explain things that, currently, nobody understands. But I guess it kind of undermines the idea of a theory-heavy approach to teaching if the theory we have can't predict the things that are actually important.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#137

Post by godelski »

WhitneyLand wrote: post #124
godelski wrote: post #99

Earlier quoted context omitted.

Is it fair to call something an introduction if it uses math from an upper division undergrad math criteria? Such as metric theory. My opinion is that it is context driven. E.g. Introduction to Differential Geometry or Introduction to Homotopy Theory. But I think you can't look at the title and infer prerequisites that are within the ballpark. I'd wager most people outside math and some physics students are familiar…

What percent of LLM knowledge requires proficiency in anything you mentioned? From what I’ve seen it’s a small percentage, and there’s no reason for most people to be put off by it. Everyone come on in the water is fine.

Between 0% and idk 70%? depending on what you're doing.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#138

Post by godelski »

WhitneyLand wrote: post #121
godelski wrote: post #97

Earlier quoted context omitted.

Don't suggest this. While I agree it can be helpful, the problem is if you're a novice you won't be able to distinguish hallucinations. Which in my experience are fairly common, especially as you do advance topice. If you got good math rigor then it's extremely helpful, because often things are hard to exactly search, but it's a potential trap for novices. But if you have no better resource, then I can't blame anyone…

That’s kind of like telling people not to go online because you can’t believe everything you read on the Internet. What proportion of the problems you’ve encountered were with the free version vs premium? It’s a huge difference and the topic here is GPT4. Also since it is fairly common for you are there any real world examples you can share?

> That’s kind of like telling people not to go online because you can’t believe everything you read on the Internet.

Uhhh... it's like telling people to trust SO over reddit, especially a subreddit known to lie.

> What proportion of the problems you’ve encountered were with the free version vs premium? It’s a huge difference and the topic here is GPT4.

Both. Can we stop doing this? This is a fairly well established principle with tons of papers written about it, especially around math. Just search arxiv, there's a new one at least every week

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#139

Post by light_hue_1 »

HybridCurve wrote: post #118
j2kun wrote: post #111

Earlier quoted context omitted.

I wrote a book: https://pimbook.org You might find it useful for your situation. The PDF is pay-what-you-want if you don't feel like paying for it.

Ah, I think I remember bookmarking this when it was posted before. You really don't have to go very far in computing to find a frontier where most everything in described pure mathematics and so it becomes a substantial barrier for undiversified autodidacts in the field. The math in these areas can often be quite advanced and difficult to approach without the proper background and so I appreciate anyone who has made…

I would suggest something like https://ocw.mit.edu/courses/6-042j-mathematics-for-computer-... instead of that book.

I appreciate that some may find the book useful, but I personally don't agree with the presentation. There are too many conceptual errors in the book that you need to unlearn to make progress. For example, the book describes R^2 as a "pair" of real numbers. This is very much untrue and that kind of thinking will lead you even further astray.

I say this as someone with a math/cs degree and PhD having taught these topics to hundreds of students.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#140

Post by 3abiton »

montyanderson wrote: post #100

for those who want some maths-heavy stuff for deep learning, check francois fluret's book https://fleuret.org/francois/lbdl.html . the pdf is free but the print is so cute.

What makes it stand out compared to OP post?
Post reply on HN