Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A non-technical explanation of deep learning

parand.com

Post reply on HN
Print view View original

61–70 of 139 posts

Re: A non-technical explanation of deep learning

#61

Post by commandlinefan »

zvmaz wrote: post #31

I have met people who think they understand a particular topic I am versed in, but actually don't. Similarly, I am often wary that I get superficial knowledge about a topic I don't know much about through "laymen" resources, and I doubt one can have an appropriate level of understanding mainly through analogies and metaphors. It's a kind of "epistemic anxiety". Of course, there are "laymen" books I stumbled upon whic…

> an appropriate level of understanding mainly through analogies and metaphors

I think it's actually worse than that - somebody who doesn't know actually realizes that he doesn't know, but somebody who _thinks_ he understands through analogies and metaphors will confidently come to the incorrect conclusion and then argue with somebody who actually does understand the topic - often managing to convince innocent bystanders because his reasoning is easier to grasp and the so-called expert seems to be getting more and more flustered (as he tries to explain why the analogy is actually correct, but oversimplified).

Re: A non-technical explanation of deep learning

#62

Post by commandlinefan »

pkdpic wrote: post #13

I love this, but Im always confused in these kinds of analogies what the reward / punishment system really equates to... Also reminds me of Ted Chiang warning us that we will torture innumerable AI entities long before we start having real conversations about treating them with compassion.

> what the reward / punishment system really equates to

Well, in the article, it says the punishment was a slap. On the other hand, he just says "she gives you a wonderful reward"... so you're left to use your imagination there.

Re: A non-technical explanation of deep learning

#63

Post by ftxbro »

amelius wrote: post #8

The problem with deep learning is opposite. You can understand most of it with just high school math. Advanced math is mostly useless because of the dimensionality of neural nets.

> Advanced math is mostly useless because of the dimensionality of neural nets.

It depends what you mean by advanced math. There is a lot of math that only really comes into play because of the high dimensionality! For example math related to tensor wrangling, low rank approximations, spectral theory, harmonic theory, matrix calculus derivatives, universality principles, and other concepts that could be interesting or bewildering or horrifying depending how you react to it. Of course some of it is only linear algebra of the 'just high school math' kind but that's not how I would normally describe it. If you look at the math in the proofs in the appendices of the more technical AI papers on arxiv there is often some weird stuff in there, not just matrix multiply and softmax.

Re: A non-technical explanation of deep learning

#64

Post by civilized »

_gmax0 wrote: post #45

The most concise and intuitive line of explanation I've been given goes along the lines of this: 1 - We want to model data, representative of some system, through functions. 2 - Virtually any function can be expressed by a n-th order polynomial. 3 - We wish to learn the parameters, the coefficients, of such polynomials. 4 - Neural networks allow us to brute-force test candidate values of such parameters (finding opti…

You don't need neural networks to do polynomial regression. Polynomial regression, perhaps surprisingly, can be implemented using only (multivariable) linear regression. You just include powers of your predictor x as terms in the regression formula:

  y = a + bx + cx^2 + dx^3 + ...
The resulting model is linear, even though there are powers of x in your formula. Because x and y are known from the data. They're not what you're solving for, you're solving for the unknown coefficients (a, b, c, d...). This gives you a linear system of equations in those unknown coefficients, which can be solved using standard linear least squares methods.

So fitting polynomials is easy. The problem is that it's not that useful. Deep learning has to solve much harder problems to get to a useful model.

Re: A non-technical explanation of deep learning

#65

Post by lhnz »

uoaei wrote: post #35
mach1ne wrote: post #18

Earlier quoted context omitted.

Yes, it’s really rather like alchemy in some sense. Stuff works, and often nobody knows exactly why.

"I don't follow the latest ML scaling and theory research" does not in any way equate to "these things are unknowable".

Hm, watching Neel Nanda videos recently and I do get the feeling that there are lots of unknowns in ML and also in what trained networks have learnt.

Re: A non-technical explanation of deep learning

#66

Post by Retric »

joe_the_user wrote: post #44
clarle wrote: post #33

Totally aware that this isn't a fully formal definition of deep learning, but one interesting takeaway for me is realizing that in a way, corporations with their formal and informal reporting structures are structured in a way similar to neural networks too. It seems like these sort of structures just regularly arise to help regulate the flow of information through a system.

Uh, The similarity of corporations and neural nets is pretty much only that both are information processing systems. An operating system or missile guidance system is far more like a corporation than a neural network. Neural networks have no memory and generally don't seek particular goals, they simply recognize, predict and generate similar instances.

Plenty of ways to think about this stuff. IMO neural networks don’t inherently do anything, it’s just a data structure.

Different ways you can interact with that data structure can however provide meaning and store information in the weights etc.

Re: A non-technical explanation of deep learning

#67

Post by lhnz »

_gmax0 wrote: post #45

The most concise and intuitive line of explanation I've been given goes along the lines of this: 1 - We want to model data, representative of some system, through functions. 2 - Virtually any function can be expressed by a n-th order polynomial. 3 - We wish to learn the parameters, the coefficients, of such polynomials. 4 - Neural networks allow us to brute-force test candidate values of such parameters (finding opti…

Hm, I don't think that's quite it. I went through my own process of learning how neural networks work recently and wrote this based on my learning: https://sebinsua.com/bridging-the-gap

As far as my understanding goes, you can represent practically any function as layers of linear transformations followed by non-linear functions (e.g. `ReLU(x) = max(0, x)`). It's this sprinkling of non-linearity that allows the networks to be able to model complex functions.

However, from my perspective, the secret sauce is (1) composability and (2) differentiability. These enable the backpropagation process (which is just "the chain rule" from calculus) and this is what allows these massive mathematical expressions to learn parameters (weights and biases) that perform well.

Re: A non-technical explanation of deep learning

#68

Post by Myrmornis »

> they see 3 spreadsheets of numbers representing the RGB values of the picture.

This needs expanding: it's the sort of thing that's easy for a programmer to say, but few non-{programmer,mathematically trained person} are going to see that an RGB value has 3 parts and so a collection of RGB values could be sliced into 3 sheets.

Re: A non-technical explanation of deep learning

#69

Post by bolyarche »

jstx1 wrote: post #36
SnooSux wrote: post #32

Earlier quoted context omitted.

I've barely forgiven him for explaining genetic algorithms and acting like they have any relevance to contemporary ML research. The footnote video was an alright explanation of backprop. If that were part of the main video that would have been reasonable. I really like his history/geography videos but anything technical leave a lot to be desired. And don't get me started on Humans Need Not Apply.

> And don't get me started on Humans Need Not Apply. Well now you have to tell us. :) Many of the concrete examples in that video are exaggerated and/or misunderstood but the general question it asks - what to do when automation makes many people unemployable through no fault of their own - seems valid.

[dead]

Re: A non-technical explanation of deep learning

#70

Post by Trufa »

jstx1 wrote: post #23

Does stuff like this help anyone? I still haven’t forgiven CGP Grey for changing the title to his 2017 ML video to “How AIs, like ChatGPT, learn”. The video is about genetic algorithms and has nothing to do with ChatGPT. (or with anything else in modern AI)

What a strange word to use in that context, why would he need to be forgiven by you? How has he wronged you? Seems at worst, an honest mistake in a complicated topic.
Post reply on HN