Is there an equivalent book/pdf for PyTorch? In general, any recommended courses/books? I've already done a first pass of fastai's 2020 course which was very eye-opening and a good segue from the introductory courses on Kaggle.
Applications of Deep Neural Networks v2 [pdf]
31–40 of 46 posts
Re: Applications of Deep Neural Networks v2 [pdf]
#32I don't like that OP is using Arxiv to upload his course material. Arxiv is supposed to be a pre-print scientific publication server. A place to post your nearly finished or ideally submitted journal or conference manuscript so you can reference it while it is being reviewed. The purpose of Arxiv is in itself a patch for a too common, too long in duration publication process. Now it is often the first place to publis…
Your criticism depends on peer-review actually being functional in the first place.
I fail to see how it does. The popularity of Arxiv is indeed a sign of a slow-functioning peer-review system, but it has its valid uses.
My criticism pertains to OP using Arxiv as a PDF host for course material irrelevant to Arxiv's userbase, i.e. expert researchers. It is already hard enough to find the quality manuscripts in Arxiv. Hence why Karpathy made Arxiv Sanity Preserver [1]. I would rather not have to drudge through pages of tutorial pdf's when searching "neural graph methods for NLP" for instance.
Re: Applications of Deep Neural Networks v2 [pdf]
#33After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
I mostly second this. Training a neural network is like training a dog, and the loss function just describes when you'll shout "no!" and how loudly. However, I believe that coding the network is very challenging unless you do a task that has been widely explored already. For optical flow, there was a wide consensus among SOTA papers for some years that convolutional filters, warping of the input data, and a hierarchi…
> Training a neural network is like training a dog, and the loss function just describes when you'll shout "no!" and how loudly.
This is a pretty good ELI5 neural networks.
Re: Applications of Deep Neural Networks v2 [pdf]
#34After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
I couldn't agree more, especially with the latter part. I've worked on action recognition with I3D for over a year now, and found that seemingly equivalent implementations in Keras, TensorFlow 2 or PyTorch will produce wildly different results. Worse yet, I found a bunch of papers that will claim SOTA results compared against one of those non-original implementations with just a few percentage-point differences. It m…
shameless ad: try mmaction2, where every result is reproducible https://github.com/open-mmlab/mmaction2 . Modelzoo: https://mmaction2.readthedocs.io/en/latest/modelzoo.html
Re: Applications of Deep Neural Networks v2 [pdf]
#35After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
What was your self study resource?
Re: Applications of Deep Neural Networks v2 [pdf]
#36Earlier quoted context omitted.
I couldn't agree more, especially with the latter part. I've worked on action recognition with I3D for over a year now, and found that seemingly equivalent implementations in Keras, TensorFlow 2 or PyTorch will produce wildly different results. Worse yet, I found a bunch of papers that will claim SOTA results compared against one of those non-original implementations with just a few percentage-point differences. It m…
shameless ad: try mmaction2, where every result is reproducible https://github.com/open-mmlab/mmaction2 . Modelzoo: https://mmaction2.readthedocs.io/en/latest/modelzoo.html
This is very cool, I’ll be studying your implementation of I3D. Did you ever attempt to train I3D end-to-end as done in the Quo Vadis paper? And it so, did you get comparable Top1/Top5 accuracy?
Re: Applications of Deep Neural Networks v2 [pdf]
#37Earlier quoted context omitted.
So far you've mentioned that you want to change three terms (loss, penalty, learning rate), one of them with a term which is already in use. You're basically rewriting the terminology to fit your personal preference. If you need to communicate with people who have experience in the field, all of this will add more confusion than it removes. It's fine if it helps you reason about things of course but it's just importa…
Nah, I’ll bend the world to my way of doing things. It’s better. Feynman had a funny story about this. I’m no Feynman, but he invented new ways of writing sin, cos, etc. He said he disliked the way it looked, since cos(x) looks like cos multiplied by x. And of course the story ended with the same punchline you outlined: when you want to talk to others, you need shared vocabulary. But the thing is, it’s extremely easy…
Yes, I think you're confused about regularisation. A regularisation is (usually) a component in the overall loss, which has the goal of simplifying or preventing overfitting, as opposed to the main component which has the goal of fitting. It's not another term or a formal term for penalty.
Re: Applications of Deep Neural Networks v2 [pdf]
#38I've been really enjoying learning Deep Learning. What are job titles where you would get to work with this? and how much knowledge is expected already?
Re: Applications of Deep Neural Networks v2 [pdf]
#39Earlier quoted context omitted.
What's wrong with the name "loss"? I like the idea of calling it "kerfuffle", but loss seems like a neutral term to me. I agree with you about fucked-up shit. I had a strong optimization background, and a traditional statistics background, and from that point everything you do with neural networks is just crazy.
We have an opportunity here to define the terms that our descendants will be using 50 years from now. It won't come again. Physics suffered from the same problem: "action," "work," and so on, are unrelated to their usage. But we're stuck with them. Both "loss" and "learning rate" are confusing, and neural networks are so confusing that I think it's worth undoing as much as possible. I would s/loss/penalty/ and s/lear…
That's really funny, 50 years from now? Backpropagation is not a law of physics, you know. Thanks for making me laugh ;).
Re: Applications of Deep Neural Networks v2 [pdf]
#40I don't like that OP is using Arxiv to upload his course material. Arxiv is supposed to be a pre-print scientific publication server. A place to post your nearly finished or ideally submitted journal or conference manuscript so you can reference it while it is being reviewed. The purpose of Arxiv is in itself a patch for a too common, too long in duration publication process. Now it is often the first place to publis…
You are around 25 years too late. Since the very beginning Arxiv has been used to publish not only cutting-edge research but also divulgation pieces, reviews, commentaries,workshop's accompanying materials,lecture notes and books.