Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Not So Mysterious or Different

arxiv.org

Post reply on HN
Print view View original

31–40 of 139 posts

Re: Deep Learning Is Not So Mysterious or Different

#31

Post by xmprt »

EncomLab wrote: post #15

The implication that any software is "mysterious" is problematic - there is no "woo" here - the exact state of the machine running the software may be determined at every cycle. The exact instruction and the data it executed with may be precisely determined, as can the next instruction. The entire mythos of any software being a "black box" is just so much advertising jargon, perpetuated by tech bros who want to belie…

The mystery was never in the "how do computers calculate the probabilities of next tokens" but rather in the "why is it able to work so well" and "what does this individual neuron contribute to the whole model"

Re: Deep Learning Is Not So Mysterious or Different

#32

Post by whiteandnerdy »

getnormality wrote: post #6

> rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. How does deep learning do this? The last time I was deeply involved in machine le…

You're correct, and the term you're looking for is "regularisation".

There are two common ways of doing this: * L1 or L2 regularisation: penalises models whose weight matrices are complex (in the sense of having lots of large elements) * Dropout: train on random subsets of the neurons to force the model to rely on simple representations that are distributed robustly across its weights

Re: Deep Learning Is Not So Mysterious or Different

#33

Post by noney4 »

hlynurd wrote: post #10
bigfatfrock wrote: post #3

Earlier quoted context omitted.

I'm a huge fan of HN just for replies such as this that smash the OP's post/product with something better. It's like at least half the reason I stick around here. Thanks for the great read.

>smash with something better Not a fan of the aggressive rhetoric here...

[dead]

Re: Deep Learning Is Not So Mysterious or Different

#34

Post by kaptainscarlet »

TechDebtDevin wrote: post #5

Anyone who wants to demystify ML should read: The StatQuest Illustrated Guide to Machine Learning [0] By Josh Starmer. To this day I haven't found a teacher who could express complex ideas as clearly and concisely as Starmer does. It's written in an almost children's book like format that is very easy to read and understand. He also just published a book on NN that is just as good. Highly recommend even if you are al…

Double Bam

Re: Deep Learning Is Not So Mysterious or Different

#35

Post by randomtoast »

rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.

Re: Deep Learning Is Not So Mysterious or Different

#36

Post by wincy »

randomtoast wrote: post #35
rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.

I’d love to see an article delve into why that is.

Re: Deep Learning Is Not So Mysterious or Different

#37

Post by EGreg »

randomtoast wrote: post #35
rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.

Saying that kind of stuff is the biggest indicator of Paul Graham (pg) himself

Re: Deep Learning Is Not So Mysterious or Different

#38

Post by ajitid »

TechDebtDevin wrote: post #5

Anyone who wants to demystify ML should read: The StatQuest Illustrated Guide to Machine Learning [0] By Josh Starmer. To this day I haven't found a teacher who could express complex ideas as clearly and concisely as Starmer does. It's written in an almost children's book like format that is very easy to read and understand. He also just published a book on NN that is just as good. Highly recommend even if you are al…

Also would like to add that he has a YouTube channel as well https://youtube.com/@statquest

Re: Deep Learning Is Not So Mysterious or Different

#39

Post by esafak »

cgdl wrote: post #2

Agreed, but PAC-Bayes or other descendants of VC theory is probably not the best explanation. The notion of algorithmic stability provides a (much) more compelling explanation. See [1] (particularly Sections 11 and 12) [1] https://arxiv.org/abs/2203.10036

Statistical mechanics is the lens that makes most sense to me, and it's well studied.

Re: Deep Learning Is Not So Mysterious or Different

#40

Post by esafak »

talles wrote: post #30

Correct me if I'm wrong, but an artificial neuron is just good old linear regression followed by an activation function to make it non linear. Make a network out of it and cool stuff happens.

MLPs are compositions of generalized linear models. That's not very enlightening though; the "mysterious" part is the macroscopics of the composition, which you can't really understand with the tools of statistics.
Post reply on HN