Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

arxiv.org

Post reply on HN
Print view View original

141–150 of 177 posts

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#141

Post by godelski »

CamperBob2 wrote: post #109
godelski wrote: post #97

Earlier quoted context omitted.

Don't suggest this. While I agree it can be helpful, the problem is if you're a novice you won't be able to distinguish hallucinations. Which in my experience are fairly common, especially as you do advance topice. If you got good math rigor then it's extremely helpful, because often things are hard to exactly search, but it's a potential trap for novices. But if you have no better resource, then I can't blame anyone…

It works better than you think, as long as you use GPT 4. See my answer to the other person ( https://news.ycombinator.com/item?id=38837646 ). A lot of negativity comes from people who goofed around with 3.X for a while, came away unimpressed, muttered something under their breath about stochastic parrots or Markov chains that sounded profound (at least to them), and never bothered to look any further. 4 is different…

That answer isn't very compelling as it is one of the most well known equations in ML. There are some very minor errors but nothing that changes the overall meaning. But you even seem to agree with me in your followup: don't rely on it, but use it. I'm only slightly stronger than you.

And stop all this 3.5 vs 4 nonesense. We all know 4 is much better. But there's plenty of literature that shows its limits, especially around memorization. You also don't understand stochastic parrots, but in fairness, seems like most people don't. LLMs start from compression algorithms and they are that at their core. But this doesn't mean it is a copy machine despite the NYT article but it also doesn't mean it is a thinking machine like the baby AGI people. Truth is in between but we can't have a real conversation because hype primed us to just bundle people into two camps and make us all true believers. Just please stop gaslighting people when they say they have run into issues. The machine is sensitive to prompts, so that can be a key difference or sometimes they might just see mistakes you don't. It's not an oracle so don't treat it like one. And don't confuse this criticism as saying LLMs suck, because I use them almost every day and love them. I just don't get why we can't be realistic about their limits and can only believe they are a golden goose or pile of shit. It's, again, neither.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#142

Post by WhitneyLand »

godelski wrote: post #138
WhitneyLand wrote: post #121

Earlier quoted context omitted.

That’s kind of like telling people not to go online because you can’t believe everything you read on the Internet. What proportion of the problems you’ve encountered were with the free version vs premium? It’s a huge difference and the topic here is GPT4. Also since it is fairly common for you are there any real world examples you can share?

> That’s kind of like telling people not to go online because you can’t believe everything you read on the Internet. Uhhh... it's like telling people to trust SO over reddit, especially a subreddit known to lie. > What proportion of the problems you’ve encountered were with the free version vs premium? It’s a huge difference and the topic here is GPT4. Both. Can we stop doing this? This is a fairly well established p…

I’ll take that as it happens so infrequently with GPT4 you have no illustrative prompts that can be shared.

There have not been tons of papers written about this.

You seem to be conflating papers about GPT4 as a solver with it as a math tutor. It’s a completely different problem space.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#143

Post by stefr- »

aurareturn wrote: post #135
tnecniv wrote: post #61

Earlier quoted context omitted.

So this is a book written by applied mathematicians for applied mathematics (they state in the preface it’s for scientists, but some theoretical scientists and engineers are essentially applied mathematics). As a result, both the topics and the presentation are biased towards those types of people. For example, I’ve never seen in practice worry about the existence and uniqueness conditions for their gradient-based op…

Are there any books you recommend for deep learning that are written for developers who don't use math every day? I suppose the goal would be to understand deep learning so that we know enough of what's going on but not to get stuck in math concepts that we probably don't know and won't use.

I am/was in this scenario. I'm sure there are other resources out there specifically aimed at developers, but a book I'm reading now is "Deep Learning From Scratch" by Seth Weidman. He takes a different approach, by explaining concepts in three distinct methods: a mathematical way, by using diagrams and by showing the code.

I like this approach because it allows me to connect the math to the problem, whereas otherwise you wouldn't have.

In the book, you're slowly creating a DL framework, as the title says, from scratch. He also has all the code on GitHub: https://github.com/SethHWeidman/DLFS_code

I think if you are truly trying to understand deep learning, you will never get to avoid the math because that's really what it is at it's core, a couple of (non-linear) functions chained together (obvious gross oversimplification).

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#144

Post by dragandj »

vivzkestrel wrote: post #116
currymj wrote: post #107

Earlier quoted context omitted.

if you have no mathematical background at all this isn’t the book for you i think. that is not really advanced mathematics although a little notationally dense. there are many good materials such as the fantastic fast.ai course that don’t require such mathematical background. if you are motivated to learn about ML, then studying the topic can gradually be a route in to more mathematical knowledge so that equations li…

all I am saying is every deep learning book I have ever opened is filled with mathematical stuff like this. I want to learn the mathematics for it but I need a starting point. Isnt there atleast one book in the entire world written with this in mind?

There is a gradual introduction: Deep Learning for Programmers (https://aiprobook.com/deep-learning-for-programmers/)

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#145

Post by go_elmo »

modeless wrote: post #136
c7b wrote: post #70

Seems like a good collection of standard ML techniques, introduced with a fairly unified mathematical notation and quite a few proofs. Quite the Herculean effort (600 pages!). It just seems to me like they're putting the emphasis on the stuff that is more straightforward to formalize rather than the stuff that would be interesting to understand. Look eg at the SGD chapter. I picked this because I think optimization i…

> they won't necessarily be much wiser about why those methods work, other than having a good intuition confirmed by numerical experiments. This is the state of the field as a whole, isn't it? > Wouldn't it be more useful to dedicate the space to convergence proofs for ADAM (which do exist) Convergence proofs don't really explain why Adam tends to work better than other methods. It's hard to blame them for not being…

*Cant predict the important things _yet_

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#146

Post by tsimionescu »

chongli wrote: post #104
ivancho wrote: post #75

Earlier quoted context omitted.

I have a strong mathematical background, and I found the notation completely insane. Right out of the gate in chapter 1 we get a definition that has subscript indices in the subscript index and a summation with subscripts in the superscript, and then composed in a giant function chain. Later we get to 4-level subscripts deep, invent at least 3 new infix operators, define 30 new symbols from 3 different alphabets and…

I’m not sure what specialization of math you studied, but using superscripts for indices is pretty common where you’re dealing with multi-dimensional objects. I used it in a lot of the courses in my degree.

They are not complaining about superscripts for indices, but about having a subscripts in those superscripts. Basically like x² but the ² has a subscript of its own. That is very dense and graphically hard to follow as notations go.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#147

Post by tsimionescu »

joshuanapoli wrote: post #27
HybridCurve wrote: post #25

As someone who has a deeper knowledge of programming rather than math, I find the mathematical notation here to be harder to understand than the code (even in a programming language I do not know). Does anyone with a stronger mathematical background here find it easier to understand the math as written more easily than the source code?

Mathematical notation usually has a problem with preferring single-letter names. We usually prefer to avoid highly abbreviated identifier names in software, because they make the program harder to read. But they’re common in Math, and I think that it makes for a lot of work jumping back and forth to remind oneself what each symbol means when trying to make sense of a statement.

I think the main difference is that in programming you typically use names from your domain, like "request" or "student". But math objects are all very abstract, they don't denote any domain. For example, if I have a triangle and I want to name its vertexes so I can refer to them later, what would be a good name? Should I call them vertexA, vertexB, and vertexC just so it's not a single letter?

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#148

Post by c7b »

modeless wrote: post #136
c7b wrote: post #70

Seems like a good collection of standard ML techniques, introduced with a fairly unified mathematical notation and quite a few proofs. Quite the Herculean effort (600 pages!). It just seems to me like they're putting the emphasis on the stuff that is more straightforward to formalize rather than the stuff that would be interesting to understand. Look eg at the SGD chapter. I picked this because I think optimization i…

> they won't necessarily be much wiser about why those methods work, other than having a good intuition confirmed by numerical experiments. This is the state of the field as a whole, isn't it? > Wouldn't it be more useful to dedicate the space to convergence proofs for ADAM (which do exist) Convergence proofs don't really explain why Adam tends to work better than other methods. It's hard to blame them for not being…

ADAM is known to have better convergence bounds than other methods. Theoretical bounds may not explain the full story of why a method works well, but it is how mathematicians reason about it. I'm only blaming them for not sharing those relevant parts of what we already know.

My bigger pain point even is how they choose to allocate their space: the theorem statements for the most relevant results are missing, the proofs for the more interesting theorems are just citations, while the proofs for basic and arguably tangentially relevant lemmata from eg probability theory take up pages and pages.

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#149

Post by QuesnayJr »

angra_mainyu wrote: post #66
youainti wrote: post #55

Earlier quoted context omitted.

As a PhD econ student, the mathematics just comes down solving constrained optimization problems. Figuring out what to consider as an optimand and the associated constraints is the real kicker.

If you're referring to micro/macro, I meant more like a mathematical introduction to the models. I recall giving Mankiw a try and wished I could just find a physics-style textbook as I found it way too wordy.

Debreu's Theory of Value

Re: Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory

#150

Post by woolion »

light_hue_1 wrote: post #139
HybridCurve wrote: post #118

Earlier quoted context omitted.

Ah, I think I remember bookmarking this when it was posted before. You really don't have to go very far in computing to find a frontier where most everything in described pure mathematics and so it becomes a substantial barrier for undiversified autodidacts in the field. The math in these areas can often be quite advanced and difficult to approach without the proper background and so I appreciate anyone who has made…

I would suggest something like https://ocw.mit.edu/courses/6-042j-mathematics-for-computer-... instead of that book. I appreciate that some may find the book useful, but I personally don't agree with the presentation. There are too many conceptual errors in the book that you need to unlearn to make progress. For example, the book describes R^2 as a "pair" of real numbers. This is very much untrue and that kind of thi…

>For example, the book describes R^2 as a "pair" of real numbers.

I naturally auto-corrected this "(the set of) pairs of real numbers". If that's the case, then I don't see how this differs from the actual definition. What is the conceptual error? Is it the missing 'set of'?

Post reply on HN