Understanding Convolution in Deep Learning (2015) • Hacker News Board

Re: Understanding Convolution in Deep Learning (2015)

#21

Post by elcritch » Wed, Sep 09, 2020, 12:15 AM UTC

Der_Einzige wrote: post #12

lacker wrote: post #10

IMO calling it "convolution" in deep learning is extra confusing, because the word "convolution" means many fairly different things in other contexts. The idea behind convolution in deep learning is that, if a particular pattern of pixels is meaningful, then it is probably also meaningful if you shift the whole thing in some direction. So you can force some layers of the network to be the same under translation, and…

You didn't explain why its faster though. It's faster because its reduces the dimensionality of the inputs down to something manageable (hundreds or low thousands). You can replace convolutions with most other types of dimensionality reduction (including other types of layers) and outside of image tasks you'll get very similar or even better performance.

I wonder if that’d work by doing a 2d Fourier transformer on an image before hand, but you’re not reducing dimensionally that way.

Re: Understanding Convolution in Deep Learning (2015)

#22

Post by bonoboTP » Wed, Sep 09, 2020, 12:59 AM UTC

lacker wrote: post #10

IMO calling it "convolution" in deep learning is extra confusing, because the word "convolution" means many fairly different things in other contexts. The idea behind convolution in deep learning is that, if a particular pattern of pixels is meaningful, then it is probably also meaningful if you shift the whole thing in some direction. So you can force some layers of the network to be the same under translation, and…

Convolution has been an established term in image processing long before convnets were invented in the 90s (or 80s or whenever). It's the same thing. It's useful to learn a bit of basic image processing, edge detection etc before jumping straight into the flashiest shiniest DL model with no basic foundations to conceptualize what is happening.

(Even before that it has been used in signal processing.)

Re: Understanding Convolution in Deep Learning (2015)

#23

Post by qppo » Wed, Sep 09, 2020, 1:19 AM UTC

elcritch wrote: post #20

qppo wrote: post #16

Earlier quoted context omitted.

I mean ultimately it comes from functional analysis and differential equations (not signal processing). It's a binary operator on functions that yields a third function. It has a lot of useful properties and equivalences, like that it can be described as the product of two Fourier transforms (although that's very roundabout). You're actually introduced to convolution in middle school when you're taught how to multipl…

It appears to be a discrete Fourier, no? Does it apply to all convolutions or just a specific instance or subset? As in id there a proof showing that as sample size N goes to a limit it approaches a continuous limit? I still natively think in continuous convolutions from Physics. The whole discretization of these operators is oddly harder for me despite it technically being simpler to compute.

No, it's true of both continuous and discrete time/domain Fourier. Convolution in time is multiplication in frequency, and vice versa. You don't need to prove this with limits directly, just use the definition of the convolution integral and Fourier transform integral.

> technically being simpler to compute.

They're equivalent, since the only meaningful way to "compute" a continuous convolution is symbolically, and discrete convolutions obey most of the same identities.

If one can place a lower bound on the time step resolution of a simulation then continuous convolutions are evaluated using discrete convolutions, which can represent the continuous case exactly via the Nyquist-Shannon sampling theorem.

Interestingly enough, to prove the Sampling Theorem you need to rely on the identity that multiplication in frequency is convolution in time, and to prove that it can't be realized in a physical system (breaks causality, since you multiply by a superposition of Heavisides which of course are infinitely long sinc functions in both directions of time).

And more interesting is that signals and systems is mostly applied dynamics and statistics, so it shouldn't be surprising if there's overlap.

Re: Understanding Convolution in Deep Learning (2015)

#24

Post by srean » Wed, Sep 09, 2020, 10:23 AM UTC

YetAnotherNick wrote: post #8

r_c_a_d wrote: post #7

Earlier quoted context omitted.

Convolution is an integration. It isn't itself a property of a domain https://en.wikipedia.org/wiki/Convolution

Try to first understand before commenting Convolution of f and g at t is: integration of f(x) * g(t - x) Cross correlation(which is termed convolution in DL) at t is: integration of f(x) * g(t + x) See the figure in the wikipedia page you shared.

I dont see why this comment was downvoted so much.