Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Applications of Deep Neural Networks v2 [pdf]

arxiv.org

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Applications of Deep Neural Networks v2 [pdf]

#41

Post by innerlee »

g_airborne wrote: post #36
innerlee wrote: post #34

Earlier quoted context omitted.

shameless ad: try mmaction2, where every result is reproducible https://github.com/open-mmlab/mmaction2 . Modelzoo: https://mmaction2.readthedocs.io/en/latest/modelzoo.html

This is very cool, I’ll be studying your implementation of I3D. Did you ever attempt to train I3D end-to-end as done in the Quo Vadis paper? And it so, did you get comparable Top1/Top5 accuracy?

Sure, checkpoints, configs and detailed training logs all are available at modelzoo https://mmaction2.readthedocs.io/en/latest/recognition_model...

The single RGB stream top1 goes up to 73.48% with resnet50, and up to 74.71% equipped with non-local. Both are much higher than the original paper with two-streams.

Re: Applications of Deep Neural Networks v2 [pdf]

#42

Post by sillysaurusx »

jmmcd wrote: post #37
sillysaurusx wrote: post #24

Earlier quoted context omitted.

Nah, I’ll bend the world to my way of doing things. It’s better. Feynman had a funny story about this. I’m no Feynman, but he invented new ways of writing sin, cos, etc. He said he disliked the way it looked, since cos(x) looks like cos multiplied by x. And of course the story ended with the same punchline you outlined: when you want to talk to others, you need shared vocabulary. But the thing is, it’s extremely easy…

Yes, I think you're confused about regularisation. A regularisation is (usually) a component in the overall loss, which has the goal of simplifying or preventing overfitting, as opposed to the main component which has the goal of fitting. It's not another term or a formal term for penalty.

Thanks for explaining that. It seems that penalty is already used, which is unfortunate. One of the interesting things about ML is that you can learn for a year and a half and still uncover more things you didn't know, which I love.

I guess I'll call loss "punishment." It matches how it feels to make progress in ML anyway.

Re: Applications of Deep Neural Networks v2 [pdf]

#43

Post by sillysaurusx »

person_of_color wrote: post #35
sillysaurusx wrote: post #8

After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…

What was your self study resource?

As glib as it sounds: pick a lot of hard projects and work on them tirelessly. Ask lots of questions on Twitter.

It was both as simple and as hard as that.

Re: Applications of Deep Neural Networks v2 [pdf]

#44

Post by jl2718 »

superbcarrot wrote: post #17
sillysaurusx wrote: post #8

After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…

> Whenever you see “loss”, substitute with “penalty” and things will become much clearer. Penalty already has a meaning in machine learning so this substitution just adds more confusion instead clarifying things. Loss seems descriptive enough to me.

“error”? “resisidual”? “objective”? “cost”? “Penalty function” is commonly used in operations research/optimization, and the parameter correction, the “penalty”. There are some conceptual differences with RL as “penalty” is more like a data input, but I think that should be a participle like “punishment” because it implies action by the trainer.

Re: Applications of Deep Neural Networks v2 [pdf]

#45

Post by thermistokles »

fxtentacle wrote: post #27
sillysaurusx wrote: post #8

After 1.5 years of self study in neural networks, my advice would be to internalize the fact that you can train a neural network to do anything that you can encode as a loss function. Networks try to minimize loss. If you want something to happen less frequently, add it to the loss. Literally addition. It was a mind-bending “there is no spoon” moment for me. Also, loss is one of the worst names imaginable. Kerfluffle…

I mostly second this. Training a neural network is like training a dog, and the loss function just describes when you'll shout "no!" and how loudly. However, I believe that coding the network is very challenging unless you do a task that has been widely explored already. For optical flow, there was a wide consensus among SOTA papers for some years that convolutional filters, warping of the input data, and a hierarchi…

> But turns out, a hierarchical structure can NOT correctly represent some movement patterns in the real world, like branches on a tree moving or overhead cables.

Could you expand a bit on this? Do you mean that it might miss small fast moving objects due to losing fidelity at the coarse resolutions. Or is there actually some sort of movement that the hierarchical structure can't interpret.

Re: Applications of Deep Neural Networks v2 [pdf]

#46

Post by fxtentacle »

thermistokles wrote: post #45
fxtentacle wrote: post #27

Earlier quoted context omitted.

I mostly second this. Training a neural network is like training a dog, and the loss function just describes when you'll shout "no!" and how loudly. However, I believe that coding the network is very challenging unless you do a task that has been widely explored already. For optical flow, there was a wide consensus among SOTA papers for some years that convolutional filters, warping of the input data, and a hierarchi…

> But turns out, a hierarchical structure can NOT correctly represent some movement patterns in the real world, like branches on a tree moving or overhead cables. Could you expand a bit on this? Do you mean that it might miss small fast moving objects due to losing fidelity at the coarse resolutions. Or is there actually some sort of movement that the hierarchical structure can't interpret.

It might miss anything where the average structure size is smaller than the large hierarchical blocks. For most SOTA, that means 32px minimum size. So it'll also miss fences, for example, because the wires are too thin and it'll not treat it as a whole but as separate tiny objects.
Post reply on HN