11–20 of 137 posts
Another bit that stood out for me:
> You’re awarded an extremely generous grant that allows you to give 200,000 people a 500-question personality test, with answers that vary on a scale from one to 10.
Dating sites like OkCupid should have such data.
Note that the writer is the famed Ingrid Daubechies, who was one of the main driver behind wavelets and early works on sparse representations.
The famous AlexNet [1] that blew away the ImageNet competition in 2012 contained 8 layers; more recent networks have even more.
Until last week [1], it was an open problem whether every function with n inputs that is computable in nondeterministic time 2^O(n) could also be computed with a two-layer circuit using only O(n) gates (that is, a deep net with just one hidden layer with a linear number of gates). This is an embarrassing state of knowledge.
Now we know the following slightly less embarrassing thing: there is an explicit function, computable in linear time, that needs at least n^3/2 gates in a two-layer deep net (modulo some log(n) factors), and another function that needs n^3/2 gates in a three-layer deep net (with the additional restriction that the output gate is a majority vote of the previous layer).
This is still a long way away from truly understanding the representational power of deep nets, but it's the first general progress that has been made since the early 90's.
It isn't really all that magical or mysterious.
> In practice, neural networks use only two or three layers... The famous AlexNet [1] that blew away the ImageNet competition in 2012 contained 8 layers; more recent networks have even more. [1] http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
> In practice, neural networks use only two or three layers... The famous AlexNet [1] that blew away the ImageNet competition in 2012 contained 8 layers; more recent networks have even more. [1] http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
Note that the writer is the famed Ingrid Daubechies, who was one of the main driver behind wavelets and early works on sparse representations.
Noticed that as well. Still writes with such reverence for the power mathematics has in unveiling the mysteries of nature. Particularly enjoyed the moniker "distinguished differential geometer" to describe Calabi! I guess its only a matter of time before "wavelet nets" are re-discovered and resurrected as the "new innovation" in machine learning ;)
Earlier quoted context omitted.
Actually the squaring was for a more practical reason. Lets try to minimize the sum of the distance, oops |y-ax| is not differentiable so we can't use calculus, lets square it instead. We later realized this worked so well because it's max likelihood of the model y=ax+b+e where e is a gaussian error function. But that came much later.
In grad school I asked that so many times trying to figure out why it was the "right" solution and this historical cheat caused so much cognitive dissonance. Then again, on the other hand, "calculus works nicely" and "it's simple" probably are good spooky directors toward useful models anyway.
> In practice, neural networks use only two or three layers... The famous AlexNet [1] that blew away the ImageNet competition in 2012 contained 8 layers; more recent networks have even more. [1] http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf
Yea it's more like the opposite, in theory you only need 1 hidden layer to get the same capacity in the network as with more hidden layers, but in practice, since auto encoders, many hidden layers gradually decreasing in size are easier to get to from local energy minimums to globally low energy minimums
[0]https://en.wikipedia.org/wiki/Universal_approximation_theore...