Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

arxiv.org

Post reply on HN
Print view View original

121–130 of 177 posts

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#121

Post by WhitneyLand »

godelski wrote: post #97
WhitneyLand wrote: post #59

Earlier quoted context omitted.

Use ChatGpt. Screenshot the math, crop it down to the equation, paste into the chat window. It can explain everything about it, what each symbol means, and how it applies to the subject. It’s an amazing accelerator for learning math. There’s no more getting stuck. I think it’s underrated because people hear “LLM’s aren’t good at math”. They are not good at certain kinds of problem solving (yet), but GPT4 is a fantast…

Don't suggest this. While I agree it can be helpful, the problem is if you're a novice you won't be able to distinguish hallucinations. Which in my experience are fairly common, especially as you do advance topice. If you got good math rigor then it's extremely helpful, because often things are hard to exactly search, but it's a potential trap for novices. But if you have no better resource, then I can't blame anyone…

That’s kind of like telling people not to go online because you can’t believe everything you read on the Internet.

What proportion of the problems you’ve encountered were with the free version vs premium? It’s a huge difference and the topic here is GPT4.

Also since it is fairly common for you are there any real world examples you can share?

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#122

Post by armoredtech »

godelski wrote: post #65
joe_the_user wrote: post #35

Earlier quoted context omitted.

If you don't have math you're going to hoodwink yourself into thinking you can get to AGI by scale alone. There are very smart people who think we can get to AGI by scale alone - they call that the "the scaling hypothesis", in fact. I think they're wrong but I thought they knew a fair amount of math. What math would you use to describe the limitations of deep learning? My impression is there aren't any exact theorems…

> "the scaling hypothesis" Oh boy, don't get me started.... I first off should say that by no means do I think any of these people (at least those publishing) are dumb. You can also be a genius in one direction and a fucking idiot in another, and that's okay. Certainly describes me haha (well less on the genius side and more on the functioning idiot side. So take everything I say with a grain of salt). Don't get me w…

ML researchers saying they need "category theory" sounds like a way to try to convince mathematicians that their work is cool. You absolutely do not need category theory.

Math is just models? Lol!

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#123

Post by chongli »

up2isomorphism wrote: post #105

[flagged]

People tend to hate things they don't understand more than the things that are wrong. The comments under reflect to some degree about this observation.

I have noticed that most HN discussions on math topics seem to devolve into complaints about notation. There seems to be a fairly large contingent of people from the programming side of things who don't have a formal background in mathematics.

I get the feeling that it isn't just frustration from a lack of understanding. If the topic was from some other technical field like organic chemistry or medicine I don't think you'd see this kind of response. I believe where the frustration comes from is adjacency: people have an expectation that they ought to be able to understand it because programming (via computer science) is adjacent to math. This expectation combined with a lack of understanding is what leads to cognitive dissonance. And that cognitive dissonance is what leads to the complaints about notation.

The notation isn't the problem. Every field has its own notation, jargon, and conventions. Mathematics notation is very simple and terse but the underlying concepts can be very abstract. Understanding comes from a lot of mathematical practice, not a glossary of terms. To paraphrase Euclid: "there is no royal road to mathematics (geometry)."

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#124

Post by WhitneyLand »

godelski wrote: post #99
ceh123 wrote: post #57

Earlier quoted context omitted.

As someone that’s in the later stages of a PhD in math, given the title starts with “Mathematical Introduction…”, the notation feels pretty reasonable for someone with a background in math. Sure I might want some slight changes to the notation I found skimming through on my phone, but everything they define and the notation they choose feels pretty familiar and I understand why they did what they did. Mirroring what…

Is it fair to call something an introduction if it uses math from an upper division undergrad math criteria? Such as metric theory. My opinion is that it is context driven. E.g. Introduction to Differential Geometry or Introduction to Homotopy Theory. But I think you can't look at the title and infer prerequisites that are within the ballpark. I'd wager most people outside math and some physics students are familiar…

What percent of LLM knowledge requires proficiency in anything you mentioned?

From what I’ve seen it’s a small percentage, and there’s no reason for most people to be put off by it.

Everyone come on in the water is fine.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#125

Post by p1esk »

jeffhwang wrote: post #87
outrun86 wrote: post #71

Earlier quoted context omitted.

Bishop’s Pattern Recognition and Machine Learning is one example that has tremendous depth and much clearer notation. Deep Learning by Goodfellow et al. is another example, albeit with less depth than Bishop. I’m glad you’re enjoying the book. The approach is ideal for a very small subset of the ML population, no doubt that was their intention. I’m just weighing in that it’s entirely possible to cover this material w…

Thanks for highlighting Bishop to me! I've self-taught through various resources esp. Goodfellow et al 2016. It's taken me a number of years to rebuild my math knowledge so that I feel comfortable with Goodfellow's treatment and look forward to learning from the Bishop book. Fwiw, I've found the math notation in the Goodfellow textbook to be among the best I've ever seen in terms of consistency and clarity. Some othe…

FYI, Bishop just released an updated DL book: https://www.bishopbook.com/

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#126

Post by p1esk »

HighFreqAsuka wrote: post #86
c7b wrote: post #74

Earlier quoted context omitted.

Neither do the authors in the book, and I'd argue that after (only) reading the book, the reader wouldn't be equipped to attempt this either (see my other post in this thread), so I think the parent poster has a point.

Yes, I have a very good point in fact. But the above comment purposely chooses not to argue with it, because it's easier to ignore it entirely and argue something else.

The problem is you presented something as a fact while it’s just a guess. Some people guess it’s an improved gradient flow, others guess it’s a smoother loss surface, someone else guesses it’s a shortcut for early layer information to reach later layers, etc. We don’t actually know why resnets work so well.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#127

Post by ivancho »

chongli wrote: post #104
ivancho wrote: post #75

Earlier quoted context omitted.

I have a strong mathematical background, and I found the notation completely insane. Right out of the gate in chapter 1 we get a definition that has subscript indices in the subscript index and a summation with subscripts in the superscript, and then composed in a giant function chain. Later we get to 4-level subscripts deep, invent at least 3 new infix operators, define 30 new symbols from 3 different alphabets and…

I’m not sure what specialization of math you studied, but using superscripts for indices is pretty common where you’re dealing with multi-dimensional objects. I used it in a lot of the courses in my degree.

I have no problem with superscripts. Here are a couple of examples of what I am talking about:

  \left(\Psi_{L} \circ \mathcal{A}_{l_{L}, l_{L-1}}^{\theta, \sum_{k=1}^{L-1} l_{k}\left(l_{k-1}+1\right)} \circ\right. & \Psi_{L-1} \circ \mathcal{A}_{l_{L-1}, l_{L-2}}^{\theta, \sum_{k=1}^{L-2} l_{k}\left(l_{k-1}+1\right)} \circ \ldots \\
  & \left.\ldots \circ \Psi_{2} \circ \mathcal{A}_{l_{2}, l_{1}}^{\theta, l_{1}\left(l_{0}+1\right)} \circ \Psi_{1} \circ \mathcal{A}_{l_{1}, l_{0}}^{\theta, 0}\right)
and

  x_{\mathcal{L}(\Psi)+k-1} & =\mathfrak{M}_{a \mathbb{1}_{(0, L)}(\mathcal{L}(\Psi)+k-1)+\mathrm{id}_{\mathbb{R}} \mathbb{1}_{\{L\}}(\mathcal{L}(\Psi)+k-1), \mathbb{D}_{k}(\Phi)}\left(\mathcal{W}_{k, \Phi} x_{\mathcal{L}(\Psi)+k-2}+\mathcal{B}_{k, \Phi}\right)
and sure, I can figure it out, but you have to agree there are some readability issues

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#129

Post by [deleted] »

two_in_one wrote: post #110

It's hard to call comprehensive. Transformers - one page. A picture would be nice. No "prompt engineering", no "double deep". In fact words "prompt" and "double" aren't used at all. "Recognition" is used only once outside of bibliography just for reference. Looks like theory will not catch up with practice any time soon. With looming singularity it's bit worrying.

[deleted]
Post reply on HN