11–20 of 46 posts
I eventually settled for Python and implemented the NN with the help of the book Make Your Own Network by Tariq Rashid. Numpy is really magic.
What book are people recommending for Deep Neural Networks? I’m working through ISLR (starting ch8) so I’ll be done in a few weeks. This topic isn’t covered so another textbook, with exercises, would be ideal. Just noticed that this paper is a book. Maybe I have a winner?
After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
I agree with you about fucked-up shit. I had a strong optimization background, and a traditional statistics background, and from that point everything you do with neural networks is just crazy.
After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
What's wrong with the name "loss"? I like the idea of calling it "kerfuffle", but loss seems like a neutral term to me. I agree with you about fucked-up shit. I had a strong optimization background, and a traditional statistics background, and from that point everything you do with neural networks is just crazy.
Physics suffered from the same problem: "action," "work," and so on, are unrelated to their usage. But we're stuck with them.
Both "loss" and "learning rate" are confusing, and neural networks are so confusing that I think it's worth undoing as much as possible.
I would s/loss/penalty/ and s/learning rate/step size/, after giving it much thought. At least, I haven't thought of better names yet.
The reason "step size" is important is because it represents what's actually going on. You don't increase the learning rate to make it learn faster. You increase the step size to make it take longer steps towards a goal. And when it's close to the goal, it circles around the goal, like water down a drain. You decrease the step size (learning rate) towards the end of training so that it doesn't keep dancing around the bowl, and can finally reach its target in the middle.
Slight modifications like that can give lots of insights. For example, now that you're thinking of learning rate in terms of water spiraling down a drain, you can see why averaging the last N model checkpoints increases accuracy: if you're spinning in a circle around a target, then the average of your last 5 positions must bring you closer to the center. In fact, that's true of any convex shape. Therefore the loss landscape seems mostly convex.
And so it goes. It's very much like compound interest. The more you understand, the more you can understand. That's why it's so important to be determined and patient.
Also, ask lots of questions on Twitter. In my opinion it's one of the most crucial resources for learning ML. The ML community there is phenomenal, and I don't know why. All I know is that everyone is super friendly and eager to help you out. Start with @pbaylies, @jonathanfly, @aydaoai, and @arfafax.
I didn't study ML in general so what I gained from the course was a deep understanding of the fundamentals & math behind it, but since I didn't get to familiarize myself with any of the existing libraries (Tensorflow/Keras back then) I had a hard time convincing anyone in industry of my skills in the field :/
Also: Why does the book only cover Deep Q-Network on Reinforcement Learning? Sure it is the most notable deep learning step in the field but, there are some relevant versions such as Actor-Critic & Maximum Entropy RL that can be very relevant too. If one includes YOLO, ResNet and newer architectures for Computer Vision application, I don't know why same things are not on RL.
Arxiv is supposed to be a pre-print scientific publication server. A place to post your nearly finished or ideally submitted journal or conference manuscript so you can reference it while it is being reviewed.
The purpose of Arxiv is in itself a patch for a too common, too long in duration publication process. Now it is often the first place to publish ML research and an obligatory source of literature in ML. No peer-review or any quality assurance makes for dubious work appearing there that can waste a lot of research time.
Hosting your elementary course there because that's where the researchers are, muddies the quality of work on Arxiv further.
After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
Penalty already has a meaning in machine learning so this substitution just adds more confusion instead clarifying things. Loss seems descriptive enough to me.
After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…
> Whenever you see “loss”, substitute with “penalty” and things will become much clearer. Penalty already has a meaning in machine learning so this substitution just adds more confusion instead clarifying things. Loss seems descriptive enough to me.
How is loss descriptive? Ah yes, we're losing... something. Our lunch, maybe.
The neural network isn't playing a game, even though people like to phrase GANs that way. There's no "win" condition. Training just ends whenever you decide to end it.
Minimizing loss doesn't bring you closer to winning a game anyway. It's often the worst strategy in certain kinds of games.
Minimizing penalty, on the other hand, is perfectly clear. If you want the neural network to do something less, add a penalty term.
I don't like that OP is using Arxiv to upload his course material. Arxiv is supposed to be a pre-print scientific publication server. A place to post your nearly finished or ideally submitted journal or conference manuscript so you can reference it while it is being reviewed. The purpose of Arxiv is in itself a patch for a too common, too long in duration publication process. Now it is often the first place to publis…
Earlier quoted context omitted.
What's wrong with the name "loss"? I like the idea of calling it "kerfuffle", but loss seems like a neutral term to me. I agree with you about fucked-up shit. I had a strong optimization background, and a traditional statistics background, and from that point everything you do with neural networks is just crazy.
We have an opportunity here to define the terms that our descendants will be using 50 years from now. It won't come again. Physics suffered from the same problem: "action," "work," and so on, are unrelated to their usage. But we're stuck with them. Both "loss" and "learning rate" are confusing, and neural networks are so confusing that I think it's worth undoing as much as possible. I would s/loss/penalty/ and s/lear…