Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning Is Not So Mysterious or Different

arxiv.org

Post reply on HN
Print view View original

41–50 of 139 posts

Re: Deep Learning Is Not So Mysterious or Different

#41

Post by Lerc »

TechDebtDevin wrote: post #5

Anyone who wants to demystify ML should read: The StatQuest Illustrated Guide to Machine Learning [0] By Josh Starmer. To this day I haven't found a teacher who could express complex ideas as clearly and concisely as Starmer does. It's written in an almost children's book like format that is very easy to read and understand. He also just published a book on NN that is just as good. Highly recommend even if you are al…

I have followed a fair few StatQuest and other videos (treadmills with Youtube are great for fitness and learning in one)

I find that no single source seems to cover things in a way that I easily understand, but cumulatively they fill in the blanks of each other.

Serrano Academy has been a good source for me as well. https://www.youtube.com/@SerranoAcademy/videos

The best tutorials give you a clear sense that the teacher has a clear understanding of the underlying principles and how/why they are applied.

I have seen a fair few things that are effectively.

'To do X, you {math thing}' While also creating the impression that they don't understand why {math thing} is the right thing to do, just that {math thing} has a name and it produces the result. Meticulously explaining the minutiae of {math thing} substitutes for a understanding of what it is doing.

It really stood out to me when looking at UMAP and seeing a bunch of things where they got into the weeds in the math without explaining why these were the particular weeds to be looking in.

Then I found a talk by Leland McInnes that had the format.

{math thing} is a tool to do {objective}. It works, there is a proof, you don't need to understand it to use the tool but the info for that is over there if you want to tale a look. These are our objectives, let's use these tools to achieve them.

The tools are neither magical black boxes, nor confused with the actual goal. It really showed the power of fully understanding the topic.

Re: Deep Learning Is Not So Mysterious or Different

#42

Post by janalsncm »

YesBox wrote: post #14

I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…

The problem is that for any reasonable value of N (>100) you will need prohibitive amounts of storage. And it will be extremely sparse. And you won’t capture any interactions between N-99 and N-98.

Transformers do that fairly well and are pretty efficient in training.

Re: Deep Learning Is Not So Mysterious or Different

#43

Post by 0cf8612b2e1e »

YesBox wrote: post #14

I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/d…

The scaling is brutal. If you have a 20k word vocabulary and want to do 3 grams, you need a 20000^3 matrix of elements (8 trillion). Most of which is going to be empty.

GPT and friends cheat by not modeling each word separately, but a large dimensional “embedding” (just a vector if you also find new vocabulary silly). The embedding represents similar words near each other in this space. The famous king-man-queen example. So even if your training set has never seen “The Queen ordered the traitor ”, it might have previously seen, “The King ordered the traitor beheaded”. The vector representation lets the model use words that represent similar concepts without concrete examples.

Re: Deep Learning Is Not So Mysterious or Different

#44

Post by EncomLab »

margalabargala wrote: post #24
EncomLab wrote: post #15

The implication that any software is "mysterious" is problematic - there is no "woo" here - the exact state of the machine running the software may be determined at every cycle. The exact instruction and the data it executed with may be precisely determined, as can the next instruction. The entire mythos of any software being a "black box" is just so much advertising jargon, perpetuated by tech bros who want to belie…

You're misunderstanding. A level of abstraction is necessary for operation of modern systems. There is no human alive who, given an intermediate step in the middle of some running learning algorithm, is able to understand and mentally model the full system at full man-made resolution, that is, down to the transistor level, on a modern CPU. Someone wishing to understand a piece of software in 2025 is forced to, at som…

It's not misunderstanding at all - but your response is certainly an attempt to obfuscate the point being made. The moment you represent anything in code, you are abstracting a real thing into it's digital representation. That digital representation if fully formed at every cycle of the digital system processing it, and the state of the system - all the way down to the transistor level may be precisely determined. To say otherwise is to make the same error as those who claim that consciousness or understanding are indefinable "extra-ordinary" things that we have to just accept exist without any justification or evidence.

Re: Deep Learning Is Not So Mysterious or Different

#45

Post by jonathanhuml »

getnormality wrote: post #6

> rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. How does deep learning do this? The last time I was deeply involved in machine le…

The solution to the L1 regularization problem is actually a specific form of the classical ReLU nonlinearity used in deep learning. I’m not sure if similar results hold for other nonlinearities, but this gave me good intuition for what thresholding is doing mathematically!

Re: Deep Learning Is Not So Mysterious or Different

#46

Post by levocardia »

whiteandnerdy wrote: post #32
getnormality wrote: post #6

> rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. How does deep learning do this? The last time I was deeply involved in machine le…

You're correct, and the term you're looking for is "regularisation". There are two common ways of doing this: * L1 or L2 regularisation: penalises models whose weight matrices are complex (in the sense of having lots of large elements) * Dropout: train on random subsets of the neurons to force the model to rely on simple representations that are distributed robustly across its weights

Dropout is roughly equivalent to layer-specific L2 regularization, and it's easy to see why: asymptotically, dropping out random neurons will achieve something similar to shrinking weights towards zero proportional to their (squared) magnitude.

Trevor Hastie's Elements of Statistical Learning has a nice proof that (for linear models) L2 regularization is also semi-equivalent to dimensionality reduction, which you could use to motivate a "simplicity prior" idea in deep learning.

Yet another way of thinking about it, in the context of ReLU units, is that a layer of ReLUs forms a truncated hyper-plane basis (like splines but in higher dimensions) in feature space, and regularization induces smoothness in this N-dimensional basis by shrinking that basis towards being a flat hyper-plane

Re: Deep Learning Is Not So Mysterious or Different

#47

Post by treyd »

wincy wrote: post #36
randomtoast wrote: post #35

Earlier quoted context omitted.

Apparently the word “delve” is the biggest indicator of the use of ChatGPT according to Paul Graham.

I’d love to see an article delve into why that is.

Because it's common in Nigerian English, which is where they outsourced a lot of the RLHF conditioning work to.

Re: Deep Learning Is Not So Mysterious or Different

#48

Post by samstave »

rottc0dd wrote: post #17

If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"( https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4... ). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear…

[dead]

Re: Deep Learning Is Not So Mysterious or Different

#49

Post by hatthew »

talles wrote: post #30

Correct me if I'm wrong, but an artificial neuron is just good old linear regression followed by an activation function to make it non linear. Make a network out of it and cool stuff happens.

This is like saying "the human brain is just some chemistry." You have the general idea correct, but there's a lot more going on that just that, and the emergent system is so much more complex that it deserves its own separate field.

Re: Deep Learning Is Not So Mysterious or Different

#50

Post by chriskanan »

getnormality wrote: post #6

> rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. How does deep learning do this? The last time I was deeply involved in machine le…

Here is an example for data-efficient vision transformers: https://arxiv.org/abs/2401.12511

Vision transformers have a more flexible hypothesis space, but they tend to have worse sample complexity than convolutional networks which have a strong architectural inductive bias. A "soft inductive bias" would be something like what this paper does where they have a special scheme for initializing vision transformers. So schemes like initialization that encourage the model to find the right solution without excessively constraining it would be a soft preference for simpler solutions.

Post reply on HN