Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Not So Mysterious or Different

arxiv.org

Post reply on HN
Print view View original

71–80 of 139 posts

Re: Deep Learning Is Not So Mysterious or Different

#71

Post by andirk »

rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

Just watched the whole thing. Thanks! I can't get in to my Masters CS: AI program at UC Berkeley because I'm dumb, but seeing this 1st day of a Probability class kinda felt like I was beginning that program haha.

I will add a great find for starting one's AI journey https://www.youtube.com/watch?v=_xIwjmCH6D4 . Kind of needs one to know intermediate CS since 1st step is "learn Python".

Re: Deep Learning Is Not So Mysterious or Different

#72

Post by nomel »

chamomeal wrote: post #53
rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

I watched the 3b1b series on neural nets years ago, and it still accounts for 95% of my understanding of AI in general. I’m not an ML person, but still. That guy has a serious gift for explaining stuff. His video on the uncertainty principle explained stuff to me that my entire undergrad education failed to!

https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_6700...

Re: Deep Learning Is Not So Mysterious or Different

#73

Post by margalabargala »

EncomLab wrote: post #66
margalabargala wrote: post #63

Earlier quoted context omitted.

Okay, then, you're just using your own personal definition of "black box" instead of the one everyone else uses. Something that's a black box is unknown to the speaker. It's not understood to be unknowable to anyone.

So your claim is that there are instructions, data, or both that are unable to be determined in what, is by definition, a fully deterministic machine?

By an individual person, yes. I claim that there exists no single human capable of fully understanding the totality of the software and hardware down to the individual transistor level.

Re: Deep Learning Is Not So Mysterious or Different

#74

Post by cratermoon »

Hugsun wrote: post #57
treyd wrote: post #47

Earlier quoted context omitted.

Because it's common in Nigerian English, which is where they outsourced a lot of the RLHF conditioning work to.

Really!? Do you have a source for this? This would be really interesting if true.

https://www.theguardian.com/technology/2024/apr/16/techscape...

Re: Deep Learning Is Not So Mysterious or Different

#75

Post by YesBox »

0cf8612b2e1e wrote: post #43
YesBox wrote: post #14

I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…

The scaling is brutal. If you have a 20k word vocabulary and want to do 3 grams, you need a 20000^3 matrix of elements (8 trillion). Most of which is going to be empty. GPT and friends cheat by not modeling each word separately, but a large dimensional “embedding” (just a vector if you also find new vocabulary silly). The embedding represents similar words near each other in this space. The famous king-man-queen exam…

You shouldn't need to allocate every possible combination !_! if you dynamically add new pairs/distance as you find them. Im talkin simple for loops.

Re: Deep Learning Is Not So Mysterious or Different

#76

Post by Vaslo »

rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

Great recommendations

Re: Deep Learning Is Not So Mysterious or Different

#77

Post by UltraSane »

EncomLab wrote: post #15

The implication that any software is "mysterious" is problematic - there is no "woo" here - the exact state of the machine running the software may be determined at every cycle. The exact instruction and the data it executed with may be precisely determined, as can the next instruction. The entire mythos of any software being a "black box" is just so much advertising jargon, perpetuated by tech bros who want to belie…

The mystery is in how the data is encoded in the parameters and why LLMs performance scales so well with parameters. The key seems to be almost orthogonal vectors that allow neural networks to store so much data. They allow 2^(cn) vectors to be learned in an n-dimensional space with c being a constant.Since almost orthogonal vectors have very small dot products, they minimally interfere with each other, allowing many concepts to coexist with limited cross talk which enables superposition

Re: Deep Learning Is Not So Mysterious or Different

#78

Post by singulargalaxy »

cgdl wrote: post #2

Agreed, but PAC-Bayes or other descendants of VC theory is probably not the best explanation. The notion of algorithmic stability provides a (much) more compelling explanation. See [1] (particularly Sections 11 and 12) [1] https://arxiv.org/abs/2203.10036

Hard disagree. Your link relies on gradient descent as an explanation, whereas OP explains why optimization is not needed to understand DL generalization. PAC-Bayes, and the other different countable hypothesis bounds in OP also are quite divergent from VC dimension. The whole point of OP seems to be that these other frameworks, unlike VC dimension, can explain generalization with an arbitrarily flexible hypothesis space.

Re: Deep Learning Is Not So Mysterious or Different

#79

Post by singulargalaxy »

superidiot1932 wrote: post #11
hlynurd wrote: post #10

Earlier quoted context omitted.

>smash with something better Not a fan of the aggressive rhetoric here...

I too felt threatened

Yeah, and it's not "better", but actually less general, relying on optimization/GD, unlike OP.

Re: Deep Learning Is Not So Mysterious or Different

#80

Post by cake-rusk »

rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

Yeah I took CS109 (through SCPD), it was a blast. But it took some serious time commitment.
Post reply on HN