Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feature Visualization: How neural nets build up their understanding of images

distill.pub

Post reply on HN
Print view View original

61–66 of 66 posts

Re: Feature Visualization: How neural nets build up their understanding of images

#61

Post by pinouchon »

colah3 wrote: post #2

Hey! I'm one of the authors, along with Alex and Ludwig. We're happy to answer any questions! :)

Hi. Thanks for the great article.

Can you elaborate on what you mean by "directions in activation space" ? If I understand it right:

You take a few neurons in a layer, and you follow some linear combination of their weights; you are then walking along a random direction. You take a single neuron and walk along its weights; you walk along this neuron's direction. Is this correct?

Also another comment: Szegedy et al [9] (https://arxiv.org/pdf/1312.6199.pdf) has the following abstract:

> First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks.

This is one of the predictions of the following paper, even though I think it came later: Opening the Black Box of Deep Neural Networks via Information (https://arxiv.org/abs/1703.00810). Here: https://youtu.be/FSfN2K3tnJU?t=1h19m23s, Tishby says: "One of the consequences of this theory is that single neurons in a layer don't tell us much."

Also, If you extend this idea to training examples, you get Mixup (https://arxiv.org/pdf/1710.09412.pdf)

Re: Feature Visualization: How neural nets build up their understanding of images

#62

Post by ludwigschubert »

radarsat1 wrote: post #34

Great presentation, but I do wish they'd throw in an equation or two. When they talk about the "channel objective", which they describe as "layer_n[:,:,z]", do they mean they are finding parameters that maximize the sum of the activations of RGB values of each channel? I'm not quite sure what the scalar loss function actually is here. I'm assuming some mean. (They discuss a few reduction operators, L_inf, L_2, in the…

You're right: we are taking the mean of the activations of a given channel `z` over all its `x,y` coordinates. (We could sum, but we use mean so that step sizes are comparable between channel and neuron objectives.) Thanks for the feedback that this notation is not super clear, we will consider rewriting those expressions.

When we do feature visualization we do start from a random point/noise. For the diagram showing steepest descent directions, however, the gradient is evaluated on an input image from the dataset, shown as the leftmost image. There's no real step size either as we're showing the direction. You can think of the scale as arbitrary and chosen for appearance.

Section numbers are on their way—and figure numbers also sound helpful! I've added a ticket. (https://github.com/distillpub/template/issues/63) For now you can already link to figures like this: https://distill.pub/2017/feature-visualization/#steepest-des...

Re: Feature Visualization: How neural nets build up their understanding of images

#64

Post by radarsat1 »

ludwigschubert wrote: post #62
radarsat1 wrote: post #34

Great presentation, but I do wish they'd throw in an equation or two. When they talk about the "channel objective", which they describe as "layer_n[:,:,z]", do they mean they are finding parameters that maximize the sum of the activations of RGB values of each channel? I'm not quite sure what the scalar loss function actually is here. I'm assuming some mean. (They discuss a few reduction operators, L_inf, L_2, in the…

You're right: we are taking the mean of the activations of a given channel `z` over all its `x,y` coordinates. (We could sum, but we use mean so that step sizes are comparable between channel and neuron objectives.) Thanks for the feedback that this notation is not super clear, we will consider rewriting those expressions. When we do feature visualization we do start from a random point/noise. For the diagram showing…

Ah, thanks for your explanation re the gradient images, I got it, thanks! I think it does say that more or less in the text actually, I was understanding it a bit wrong, my bad. For me this preconditoning part of the article is the hardest to get an intuition for.

Re: Feature Visualization: How neural nets build up their understanding of images

#65

Post by jasonszhao »

muxator wrote: post #16

Looking at the finger instead of the moon: I like the HTML layout (responsive, inline images with captions, lateral notes). Any insights on how it's generated? Markdown, Rst, Latex -> HTML? I would love to produce my documentation in this way. Edit: I was too hurried. Everything is explained in https://distill.pub/guide/ , the template is at https://github.com/distillpub/template

The page is broken on Safari 10.0 but works on Chrome 59

Re: Feature Visualization: How neural nets build up their understanding of images

#66

Post by michaf »

Kronopath wrote: post #45

Interesting that simple optimization ends up with high-frequency noise similar to adversarial attacks on neural nets. While I agree that the practicality of these visualizations mean that you have to fight against this high-frequency "cheating", I can't help but shake the feeling that what these optimization visualizations are showing us is correct . This is what the neuron responds to, whether you like it or not. Pu…

The article mentions pooling layers as one source for the high frequency patterns. Jeffrey Hinton recently introduced capsule networks (https://news.ycombinator.com/item?id=15609402), in part because he wants to get rid of pooling layers. Maybe this approach is effective to counter (at least visually indistinguishable) adversarial examples.

edit: ok, someone already tested it, and it does not seem to help that much: https://github.com/jaesik817/adv_attack_capsnet

Post reply on HN