Applications of Deep Neural Networks v2 [pdf]
1–10 of 46 posts
Re: Applications of Deep Neural Networks v2 [pdf]
#2Re: Applications of Deep Neural Networks v2 [pdf]
#3Same guy? https://www.youtube.com/user/HeatonResearch
Yes, looks like the same person. The Code tab on arxiv page points to the same github account as on the youtube page.
Re: Applications of Deep Neural Networks v2 [pdf]
#4What book are people recommending for Deep Neural Networks?
I’m working through ISLR (starting ch8) so I’ll be done in a few weeks.
This topic isn’t covered so another textbook, with exercises, would be ideal.
Just noticed that this paper is a book. Maybe I have a winner?
Re: Applications of Deep Neural Networks v2 [pdf]
#5What book are people recommending for Deep Neural Networks? I’m working through ISLR (starting ch8) so I’ll be done in a few weeks. This topic isn’t covered so another textbook, with exercises, would be ideal. Just noticed that this paper is a book. Maybe I have a winner?
This paper seems really interesting, and it’s geared towards applications I guess
For more fundamental material I like the CS231n course notes [0] and Goodfellow, Bengio, and Courville [1]
Re: Applications of Deep Neural Networks v2 [pdf]
#6What book are people recommending for Deep Neural Networks? I’m working through ISLR (starting ch8) so I’ll be done in a few weeks. This topic isn’t covered so another textbook, with exercises, would be ideal. Just noticed that this paper is a book. Maybe I have a winner?
Have been hearing good things about this - http://d2l.ai/
Have to get to it soon.
Re: Applications of Deep Neural Networks v2 [pdf]
#7What book are people recommending for Deep Neural Networks? I’m working through ISLR (starting ch8) so I’ll be done in a few weeks. This topic isn’t covered so another textbook, with exercises, would be ideal. Just noticed that this paper is a book. Maybe I have a winner?
The Deep Learning book is great & I have a copy, but its not honestly something that I read cover to cover.
To me, Its more of a reference book.
But if you want a "Fyneman" type book that describes the underlying structure & workings in a non academic way, I would recommend;
Michael Nielsen's Neural Networks & Deep Learning[0]
Jeff Heaton's Introduction To The Math Of Neural Networks[1]
[0]:http://neuralnetworksanddeeplearning.com/index.html
[1]:https://www.amazon.com/Introduction-Math-Neural-Networks-Hea...
Re: Applications of Deep Neural Networks v2 [pdf]
#8After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function.
Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition.
It was a mind-bending “there is no spoon” moment for me.
Also, loss is one of the worst names imaginable. Kerfluffle would’ve been better, because at least it’s mostly meaningless. Whenever you see “loss”, substitute with “penalty” and things will become much clearer.
Secondary advice: if you don’t have patience, you won’t get anywhere. In the same way that the stock market is a lever for transferring money from the impatient to the patient, neural networks are a lever to transfer advantage to the patient.
By that I mean, I can’t count the number of times I almost wrote off some small tweak as “doesn’t work”, only to leave the network training for another week or so and discovering it worked fine. In fact, it was almost always equivalent, or had no advantage, I.e. a placebo. It’s not like code; you can do so much fucked-up shit to a neural network, and it will still work. It’s unlike anything you’re used to.
Beyond that, just remember that this stuff is hard. Coding the network is easy. Getting it right is hard. And getting it perfect, well, took me a year. Google’s official biggan model at google/compare_gan never achieved the same FID as real biggan. Why? I immersed myself in this mystery, eventually reverse engineering the official tensorflow graph. I discovered their implementation was missing a crucial + 1, so their gamma was centered around zero instead of one. And in batchnorm, gamma is a multiplier — so the network was basically multiplied by zero and no one noticed for years. (Remember how I said you can do a lot to a network without causing problems? Sometimes the problems are so subtle they’ll drive you nuts. You know something is wrong, but you don’t know what or why, and it’s almost impossible to debug.)
Re: Applications of Deep Neural Networks v2 [pdf]
#9After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
I’d be interested in reading your formal write up. It’d be interesting to see the issue as you understand it.
Re: Applications of Deep Neural Networks v2 [pdf]
#10After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
I’d be interested in reading your formal write up. It’d be interesting to see the issue as you understand it.
Sure thing! https://github.com/google/compare_gan/issues/54
It’s not much of a writeup. It’s basically saying, hey, this is zero when it should be one.
The results were dramatic. It went from blobs to replicating the biggan paper almost perfectly. I think we’re at a FID of 11 or so on imagenet. Here's a screenshot I just pulled from our current run: https://i.imgur.com/k1RuWEG.png
Stole a year of my life to track it down. But it was a puzzle I couldn’t put down. It haunted my dreams. I was tossing and turning like, but why won’t it work... why won’t it work...