11–20 of 24 posts
IMO calling it "convolution" in deep learning is extra confusing, because the word "convolution" means many fairly different things in other contexts. The idea behind convolution in deep learning is that, if a particular pattern of pixels is meaningful, then it is probably also meaningful if you shift the whole thing in some direction. So you can force some layers of the network to be the same under translation, and…
IMO calling it "convolution" in deep learning is extra confusing, because the word "convolution" means many fairly different things in other contexts. The idea behind convolution in deep learning is that, if a particular pattern of pixels is meaningful, then it is probably also meaningful if you shift the whole thing in some direction. So you can force some layers of the network to be the same under translation, and…
It's faster because its reduces the dimensionality of the inputs down to something manageable (hundreds or low thousands). You can replace convolutions with most other types of dimensionality reduction (including other types of layers) and outside of image tasks you'll get very similar or even better performance.
Published in 2015
Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
"At lower levels, convolutional nets will not perform cross correlation, because we know that they perform edge detection in the very first convolutional layers. But in later layers, where more abstract features are generated, it is possible that a convolutional net learns to perform cross-correlation by convolution. It is imaginable that the bright pixels from the cross-correlation will be redirected to units which detect faces (the Google brain project has some units in its architecture which are dedicated to faces, cats etc.; maybe cross correlation plays a role here?)."
Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
The terminology comes from signal processing, where a convolution in the frequency domain is equivalent to a multiplication in the time domain. I don't think anyone is thinking about the frequency domain in deep-learning, but they still call the operators convolution kernels.
It's a binary operator on functions that yields a third function. It has a lot of useful properties and equivalences, like that it can be described as the product of two Fourier transforms (although that's very roundabout).
You're actually introduced to convolution in middle school when you're taught how to multiply monomials to build a polynomial (at my middle school they called it "FOIL").
Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
That doesn't help one understand what it is at all. Convolution in DL is simply a set of dot products of a patch from the input with a bunch of filters. Each resulting dot product is simply a measure of similarity between a patch and a filter. That's all there is to it.
Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
Earlier quoted context omitted.
Multiplying in frequency domain is convolution, in DL terminology convolution is that convolution with the weights rotated by 180 degree.
Convolution is an integration. It isn't itself a property of a domain https://en.wikipedia.org/wiki/Convolution
1 : a form or shape that is folded in curved or tortuous windings e.g the convolutions of the intestines
2 : one of the irregular ridges on the surface of the brain and especially of the cerebrum of higher mammals
3 : a complication or intricacy of form, design, or structure … societies in which the convolutions of power and the caprices of the powerful are ever-present dangers to survival.
After this is clear read the mathematical idea on wikipedia. After reading that, do google scholar search on the AI papers that first mentioned it. That is the way to go.
Earlier quoted context omitted.
The terminology comes from signal processing, where a convolution in the frequency domain is equivalent to a multiplication in the time domain. I don't think anyone is thinking about the frequency domain in deep-learning, but they still call the operators convolution kernels.
I mean ultimately it comes from functional analysis and differential equations (not signal processing). It's a binary operator on functions that yields a third function. It has a lot of useful properties and equivalences, like that it can be described as the product of two Fourier transforms (although that's very roundabout). You're actually introduced to convolution in middle school when you're taught how to multipl…