Applications of Deep Neural Networks v2 [pdf] • Hacker News Board

Re: Applications of Deep Neural Networks v2 [pdf]

#21

Post by sillysaurusx » Mon, Jan 25, 2021, 9:14 AM UTC

notretarded wrote: post #20

sillysaurusx wrote: post #14

Earlier quoted context omitted.

We have an opportunity here to define the terms that our descendants will be using 50 years from now. It won't come again. Physics suffered from the same problem: "action," "work," and so on, are unrelated to their usage. But we're stuck with them. Both "loss" and "learning rate" are confusing, and neural networks are so confusing that I think it's worth undoing as much as possible. I would s/loss/penalty/ and s/lear…

It's not penalty or step size. It's loss as in amount of information lossd (not encoded in your network) compared to one perfectly encoding ground truth. Learning rate, as in what is the maximum amount of delta you are allowed to change your inputs to minimise your information loss analogous to how quickly you can possibly learn in one experiment.

Fair. I’ll keep that in mind. On the other hand, it went way over my head, and I’m not afraid to admit it.

One of the nice things about ML (and math, for that matter) is that there are multiple mathematically equivalent ways of looking at a thing.

Re: Applications of Deep Neural Networks v2 [pdf]

#22

Post by [deleted] » Mon, Jan 25, 2021, 9:16 AM UTC

gillesjacobs wrote: post #16

I don't like that OP is using Arxiv to upload his course material. Arxiv is supposed to be a pre-print scientific publication server. A place to post your nearly finished or ideally submitted journal or conference manuscript so you can reference it while it is being reviewed. The purpose of Arxiv is in itself a patch for a too common, too long in duration publication process. Now it is often the first place to publis…

[deleted]

Re: Applications of Deep Neural Networks v2 [pdf]

#23

Post by superbcarrot » Mon, Jan 25, 2021, 9:17 AM UTC

sillysaurusx wrote: post #18

superbcarrot wrote: post #17

Earlier quoted context omitted.

> Whenever you see “loss”, substitute with “penalty” and things will become much clearer. Penalty already has a meaning in machine learning so this substitution just adds more confusion instead clarifying things. Loss seems descriptive enough to me.

Perhaps, but regularization is a better name for that term anyway. How is loss descriptive? Ah yes, we're losing... something. Our lunch, maybe. The neural network isn't playing a game, even though people like to phrase GANs that way. There's no "win" condition. Training just ends whenever you decide to end it. Minimizing loss doesn't bring you closer to winning a game anyway. It's often the worst strategy in certain…

So far you've mentioned that you want to change three terms (loss, penalty, learning rate), one of them with a term which is already in use. You're basically rewriting the terminology to fit your personal preference. If you need to communicate with people who have experience in the field, all of this will add more confusion than it removes. It's fine if it helps you reason about things of course but it's just important to keep in mind that the rest of the world isn't on board.

Re: Applications of Deep Neural Networks v2 [pdf]

#24

Post by sillysaurusx » Mon, Jan 25, 2021, 9:25 AM UTC

superbcarrot wrote: post #23

sillysaurusx wrote: post #18

Earlier quoted context omitted.

Perhaps, but regularization is a better name for that term anyway. How is loss descriptive? Ah yes, we're losing... something. Our lunch, maybe. The neural network isn't playing a game, even though people like to phrase GANs that way. There's no "win" condition. Training just ends whenever you decide to end it. Minimizing loss doesn't bring you closer to winning a game anyway. It's often the worst strategy in certain…

So far you've mentioned that you want to change three terms (loss, penalty, learning rate), one of them with a term which is already in use. You're basically rewriting the terminology to fit your personal preference. If you need to communicate with people who have experience in the field, all of this will add more confusion than it removes. It's fine if it helps you reason about things of course but it's just importa…

Nah, I’ll bend the world to my way of doing things. It’s better.

Feynman had a funny story about this. I’m no Feynman, but he invented new ways of writing sin, cos, etc. He said he disliked the way it looked, since cos(x) looks like cos multiplied by x. And of course the story ended with the same punchline you outlined: when you want to talk to others, you need shared vocabulary.

But the thing is, it’s extremely easy to remember to say “loss” instead of “penalty” when I’m talking to someone. But it was extremely hard for me to even understand what the heck a loss was. What is it, exactly? What’s it doing and why? How should I think about it — and more importantly, how can I extrapolate that thinking to take advantage of it?

Maybe it’s a personal quirk, but I simply couldn’t understand loss. I know penalty though. Ditto tor learning rate vs step size. So it’s more of “internal advice” rather than me saying that you should rewrite your papers with the new names.

EDIT: By the way, I wasn't proposing that "penalty be renamed to "regularization". I was under the impression that what the parent comment was calling penalty" was normally called "regularization", i.e. that regularization was the formal name for it. If that's not true, it's possible my understanding is incomplete -- what is penalty? I haven't heard of it till now, to be honest. And googling for "machine learning penalty" pops up 5 articles on regularization.

So I was proposing two changes: loss -> penalty, and learning rate -> step size.

Re: Applications of Deep Neural Networks v2 [pdf]

#25

Post by dragandj » Mon, Jan 25, 2021, 9:29 AM UTC

If I may drop in with a bit of shameless self-promotion.

My "Deep Learning for Programmers: A Tutorial with CUDA, OpenCL, DNNL, Java, and Clojure" book explains and executes every single line of code interactively, from low level operations to high-level networks that do everything automatically. The code is built on the state of the art performance operations of oneDNN (Intel, CPU) and cuDNN (CUDA, GPU). Very concise readable and understandable by humans.

https://aiprobook.com/deep-learning-for-programmers/

Here's the open source library built throughout the book:

https://github.com/uncomplicate/deep-diamond

Some chapters from the beginning of the book are available on my blog, as a tutorial series:

https://dragan.rocks

Re: Applications of Deep Neural Networks v2 [pdf]

#26

Post by tpoacher » Mon, Jan 25, 2021, 9:33 AM UTC

superbcarrot wrote: post #23

sillysaurusx wrote: post #18

Earlier quoted context omitted.

Perhaps, but regularization is a better name for that term anyway. How is loss descriptive? Ah yes, we're losing... something. Our lunch, maybe. The neural network isn't playing a game, even though people like to phrase GANs that way. There's no "win" condition. Training just ends whenever you decide to end it. Minimizing loss doesn't bring you closer to winning a game anyway. It's often the worst strategy in certain…

So far you've mentioned that you want to change three terms (loss, penalty, learning rate), one of them with a term which is already in use. You're basically rewriting the terminology to fit your personal preference. If you need to communicate with people who have experience in the field, all of this will add more confusion than it removes. It's fine if it helps you reason about things of course but it's just importa…

Just because terms become established doesn't mean the way they got about becoming established was through clarity and careful deliberation. In fact I'd go as far as saying that more than half of the terms/notation in such fields sound like they were created as silly placeholder names which then stuck. So much so that we need a translation of their actual meaning each time they're used. Even something as basic as p(x).

Re: Applications of Deep Neural Networks v2 [pdf]

#27

Post by fxtentacle » Mon, Jan 25, 2021, 9:36 AM UTC

sillysaurusx wrote: post #8

After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…

I mostly second this. Training a neural network is like training a dog, and the loss function just describes when you'll shout "no!" and how loudly.

However, I believe that coding the network is very challenging unless you do a task that has been widely explored already. For optical flow, there was a wide consensus among SOTA papers for some years that convolutional filters, warping of the input data, and a hierarchical structure was the correct way, e.g. everything descended from FlowNet.

But turns out, a hierarchical structure can NOT correctly represent some movement patterns in the real world, like branches on a tree moving or overhead cables. So now we have a category of AI solutions that all fail in the same way in the same circumstances, plus commercial products (e.g. Skydio Drone) with the exact same issues.

The correct approach seems to be a iterative solver approach, which has been attempted with RAFT, but nobody has yet managed to design a suitable network architecture that does not require hierarchical undersampling.

Just like in your failure story, tiny mistakes in the network can prevent success for good. And you need lots of attention to detail and plenty of experience to avoid those mistakes.

Re: Applications of Deep Neural Networks v2 [pdf]

#28

Post by pugio » Mon, Jan 25, 2021, 9:57 AM UTC

tpoacher wrote: post #26

superbcarrot wrote: post #23

Earlier quoted context omitted.

So far you've mentioned that you want to change three terms (loss, penalty, learning rate), one of them with a term which is already in use. You're basically rewriting the terminology to fit your personal preference. If you need to communicate with people who have experience in the field, all of this will add more confusion than it removes. It's fine if it helps you reason about things of course but it's just importa…

Just because terms become established doesn't mean the way they got about becoming established was through clarity and careful deliberation. In fact I'd go as far as saying that more than half of the terms/notation in such fields sound like they were created as silly placeholder names which then stuck. So much so that we need a translation of their actual meaning each time they're used. Even something as basic as p(x…

I still feel mental friction when contemplating anything to do with "regression" because the word doesn't seem to capture what the technique(s) (e.g. linear, logistic) actually do.

I have looked into the historical context and reason for the use of the word (the technique was first popularized in something which "regressed to the mean"), as well as its development, and it still bugs me any time.

Re: Applications of Deep Neural Networks v2 [pdf]

#29

Post by hoseja » Mon, Jan 25, 2021, 10:05 AM UTC

gillesjacobs wrote: post #16

I don't like that OP is using Arxiv to upload his course material. Arxiv is supposed to be a pre-print scientific publication server. A place to post your nearly finished or ideally submitted journal or conference manuscript so you can reference it while it is being reviewed. The purpose of Arxiv is in itself a patch for a too common, too long in duration publication process. Now it is often the first place to publis…

Your criticism depends on peer-review actually being functional in the first place.

Re: Applications of Deep Neural Networks v2 [pdf]

#30

Post by g_airborne » Mon, Jan 25, 2021, 10:15 AM UTC

sillysaurusx wrote: post #8

After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…

I couldn't agree more, especially with the latter part. I've worked on action recognition with I3D for over a year now, and found that seemingly equivalent implementations in Keras, TensorFlow 2 or PyTorch will produce wildly different results. Worse yet, I found a bunch of papers that will claim SOTA results compared against one of those non-original implementations with just a few percentage-point differences. It makes no sense! It took me hundreds of hours to hunt down the differences between how these frameworks implement their layers before I could come even close to the expected accuracy...