Understanding Convolution in Deep Learning (2015)
timdettmers.com
Understanding Convolution in Deep Learning (2015)
1–10 of 24 posts
Re: Understanding Convolution in Deep Learning (2015)
#2Published in 2015
Re: Understanding Convolution in Deep Learning (2015)
#3Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
Re: Understanding Convolution in Deep Learning (2015)
#4Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
As it is mostly done with weights whose initialization and any operation will be same with the flipping, it can basically be imagined whatever you find easy to imagine.
Re: Understanding Convolution in Deep Learning (2015)
#5Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
The terminology comes from signal processing, where a convolution in the frequency domain is equivalent to a multiplication in the time domain. I don't think anyone is thinking about the frequency domain in deep-learning, but they still call the operators convolution kernels.
Re: Understanding Convolution in Deep Learning (2015)
#6Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
The terminology comes from signal processing, where a convolution in the frequency domain is equivalent to a multiplication in the time domain. I don't think anyone is thinking about the frequency domain in deep-learning, but they still call the operators convolution kernels.
Multiplying in frequency domain is convolution, in DL terminology convolution is that convolution with the weights rotated by 180 degree.
Re: Understanding Convolution in Deep Learning (2015)
#7Earlier quoted context omitted.
The terminology comes from signal processing, where a convolution in the frequency domain is equivalent to a multiplication in the time domain. I don't think anyone is thinking about the frequency domain in deep-learning, but they still call the operators convolution kernels.
Multiplying in frequency domain is convolution, in DL terminology convolution is that convolution with the weights rotated by 180 degree.
Convolution is an integration. It isn't itself a property of a domain https://en.wikipedia.org/wiki/Convolution
Re: Understanding Convolution in Deep Learning (2015)
#8Earlier quoted context omitted.
Multiplying in frequency domain is convolution, in DL terminology convolution is that convolution with the weights rotated by 180 degree.
Convolution is an integration. It isn't itself a property of a domain https://en.wikipedia.org/wiki/Convolution
Try to first understand before commenting
Convolution of f and g at t is:
integration of f(x) * g(t - x)
integration of f(x) * g(t + x)
Re: Understanding Convolution in Deep Learning (2015)
#9Everyone must first get over the terminology confusion. Convolution in DL is actually cross-correlation, not convolution. In practise it does not matter, signal is just flipped, but it can be very confusing when you try to learn and go trough examples.
The terminology comes from signal processing, where a convolution in the frequency domain is equivalent to a multiplication in the time domain. I don't think anyone is thinking about the frequency domain in deep-learning, but they still call the operators convolution kernels.
Fuck the frequency domain, here-
"Convolution with a kernel K" describes a system whose impulse response is K. In discrete time, suppose you have K=[1,2] and convolve [0,1,2,0] with it- you wind up with [0,1,3,2,0], if I'm awake enough for arithmetic.
Correlation with a kernel K is convolution with K time-reversed (i.e. [2,1])- you'd get [0,2,5,2,0] (again if I'm awake). Note that 5- right there, the input signal "lines up just right" with the kernel- 2x2 + 1x1. That's why it's called correlation- its output is big when the input looks like the kernel.
Re: Understanding Convolution in Deep Learning (2015)
#10IMO calling it "convolution" in deep learning is extra confusing, because the word "convolution" means many fairly different things in other contexts.
The idea behind convolution in deep learning is that, if a particular pattern of pixels is meaningful, then it is probably also meaningful if you shift the whole thing in some direction. So you can force some layers of the network to be the same under translation, and it'll be faster to pick up some sorts of patterns.