Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Not So Mysterious or Different

arxiv.org

Post reply on HN
Print view View original

91–100 of 139 posts

Re: Deep Learning Is Not So Mysterious or Different

#91

Post by [deleted] »

rottc0dd wrote: post #90
rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

From, a comment I posted elsewhere for written versions. There is a course reader for CS109 [1]. You can download pdf version of this. There is also book[2] for excellent caltech course[3]. [1] https://chrispiech.github.io/probabilityForComputerScientist ... [2] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/ ... [3] https://work.caltech.edu/telecourse

[deleted]

Re: Deep Learning Is Not So Mysterious or Different

#92

Post by eru »

chamomeal wrote: post #53
rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

I watched the 3b1b series on neural nets years ago, and it still accounts for 95% of my understanding of AI in general. I’m not an ML person, but still. That guy has a serious gift for explaining stuff. His video on the uncertainty principle explained stuff to me that my entire undergrad education failed to!

It helps that 3b1b doesn't start with a curriculum and then has to figure out how to teach it. Instead he can select topics to suit his style.

Re: Deep Learning Is Not So Mysterious or Different

#93

Post by roenxi »

hatthew wrote: post #49
talles wrote: post #30

Correct me if I'm wrong, but an artificial neuron is just good old linear regression followed by an activation function to make it non linear. Make a network out of it and cool stuff happens.

This is like saying "the human brain is just some chemistry." You have the general idea correct, but there's a lot more going on that just that, and the emergent system is so much more complex that it deserves its own separate field.

Although with extra irony. "linear regression followed by an activation function to make it non linear". So it isn't good old linear regression because it is explicitly delinearised.

Re: Deep Learning Is Not So Mysterious or Different

#94

Post by smus »

getnormality wrote: post #6

> rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. How does deep learning do this? The last time I was deeply involved in machine le…

the AdamW optimizer (basically the default in DL nowadays) is doing exactly that

Re: Deep Learning Is Not So Mysterious or Different

#95

Post by stevesimmons »

rottc0dd wrote: post #90
rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

From, a comment I posted elsewhere for written versions. There is a course reader for CS109 [1]. You can download pdf version of this. There is also book[2] for excellent caltech course[3]. [1] https://chrispiech.github.io/probabilityForComputerScientist ... [2] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/ ... [3] https://work.caltech.edu/telecourse

Your first two links don't work

Re: Deep Learning Is Not So Mysterious or Different

#96

Post by gnabgib »

stevesimmons wrote: post #95
rottc0dd wrote: post #90

Earlier quoted context omitted.

From, a comment I posted elsewhere for written versions. There is a course reader for CS109 [1]. You can download pdf version of this. There is also book[2] for excellent caltech course[3]. [1] https://chrispiech.github.io/probabilityForComputerScientist ... [2] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/ ... [3] https://work.caltech.edu/telecourse

Your first two links don't work

That's because they posted them somewhere else (easy mistake to make.. HN doesn't show you the full link in a comment, so copy/paste just copies the ellipsis)

https://chrispiech.github.io/probabilityForComputerScientist...

https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/...

Re: Deep Learning Is Not So Mysterious or Different

#97

Post by yomritoyj »

d_burfoot wrote: post #68

DNNs do not have special generalization powers. If anything, their generalization is likely weaker than more mathematically principled techniques like the SVM. If you try to train a DNN to solve a classical ML problem like the "Wine Quality" dataset from the UCI Machine Learning repo [0], you will get abysmal results and overfitting. The "magic" of LLMs comes from the training paradigm. Because the optimization is wo…

You may want to look at this. Neural network models with enough capacity to memorize random labels are still capable of generalizing well when fed actual data

Zhang et al (2021) 'Understanding deep learning (still) requires rethinking generalization'

https://dl.acm.org/doi/10.1145/3446776

Re: Deep Learning Is Not So Mysterious or Different

#98

Post by salviati »

chamomeal wrote: post #53
rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

I watched the 3b1b series on neural nets years ago, and it still accounts for 95% of my understanding of AI in general. I’m not an ML person, but still. That guy has a serious gift for explaining stuff. His video on the uncertainty principle explained stuff to me that my entire undergrad education failed to!

> That guy has a serious gift for explaining stuff

I'd like to challenge this idea.

I don't believe he's more gifted than other people. I strongly believe that the point is he spent a lot of time and effort to get better at explaining stuff.

He contemplated feedback and improved his explanations throughout the years.

His videos are excellent because he poured himself into making them excellent, not because he has a gift.

In my experience the professors who lack this ability do so because they don't put enough effort into it, not because they were born without it.

Re: Deep Learning Is Not So Mysterious or Different

#99

Post by procaryote »

nickysielicki wrote: post #26
YesBox wrote: post #14

I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…

Markov chains are very very far off from gpt2.

Aren't they technically the same? GPT picks the next token given the state of current context, based on probabilities and a random factor. That is mathematically equivalent to a Markov chain, isn't it?
Post reply on HN