Fooling Neural Networks [pdf] • Hacker News Board

Re: Fooling Neural Networks [pdf]

#31

Post by nestorD » Fri, Aug 06, 2021, 8:03 PM UTC

zwieback wrote: post #9

I have a background in classic image processing and machine vision and back in the olden days we had the opposite problem: algorithms were just too specific to build useful applications. It's easy to detect lines and circles with a Hough Transform or do template matching for features that very closely match a sample. However, working up the chain it never came together, detecting cars in a parking lot, a relatively s…

My feeling is that:

- lots of people in the DNN for machine vision community do not have a background in classical techniques.

- a lot of classical techniques and preprocessing pass make no real difference when applied to the input of a DNN and are thus worth eliminating from the pipeline to simplify it (this has been my experience).

However, I do think that there are gain to be gotten by combining classical image processing ideas with neural networks. It just hasn't really happened yet.

Re: Fooling Neural Networks [pdf]

#32

Post by [deleted] » Fri, Aug 06, 2021, 8:26 PM UTC

ac42dgu wrote: post #17

tarxzvf wrote: post #10

Earlier quoted context omitted.

What exactly are the right priors for general intelligence? And keep in mind, whichever prior you choose, I can design learning problem where it will lead you astray. This paper provides some interesting results on the weakness inherent in universal priors: https://arxiv.org/abs/1510.04931

Related question: What are the adversarial examples for human intelligence? We know some for the visual and auditory systems, but what about the arguably general intelligence of humans? Maybe we can work our way backwards from the adversarial examples to the inductive biases?

[deleted]

Re: Fooling Neural Networks [pdf]

#33

Post by omeze » Fri, Aug 06, 2021, 8:26 PM UTC

zwieback wrote: post #9

I have a background in classic image processing and machine vision and back in the olden days we had the opposite problem: algorithms were just too specific to build useful applications. It's easy to detect lines and circles with a Hough Transform or do template matching for features that very closely match a sample. However, working up the chain it never came together, detecting cars in a parking lot, a relatively s…

There's some stuff happening, eg applying anti-aliasing to improve shift invariance: https://richzhang.github.io/antialiased-cnns/ (also check out the related papers)

Re: Fooling Neural Networks [pdf]

#34

Post by AareyBaba » Fri, Aug 06, 2021, 8:31 PM UTC

only_as_i_fall wrote: post #8

Anyone have a sense of how much of a problem this is? It's not surprising that a network can be fooled by small input changes, but if some image preprocessing is enough to solve this It's not a big problem. On the other hand, if I can make a sign that looks like a stop sign to people but looks like a road work sign to a tesla, that's obviously a big deal. These slides touch on the difference by saying that physical e…

Tesla identifying the moon as a yellow traffic light ? https://twitter.com/JordanTeslaTech/status/14184133078625853...

Re: Fooling Neural Networks [pdf]

#35

Post by c0deb0t » Fri, Aug 06, 2021, 8:44 PM UTC

Adversarial attacks is a super interesting field, but unfortunately I feel that a lot of papers are just incremental attack or defense improvements like a cat-and-mouse game. I originally did some research on 3D point cloud attacks, but later stopped because making super successful attacks (eg., attacks with higher success rates than all the previous techniques for some very specific task) don't really help us understand that much more about neural nets, its just optimizing a metric for publishing papers. This kind of research is quite common, even at top conferences.

Despite this, recently, we made a 1 minute explainer video introducing adversarial attacks on neural nets as a submission for the Veritasium contest: https://youtu.be/hNuhdf-fL_g Give it a watch!

Re: Fooling Neural Networks [pdf]

#36

Post by ballenf » Fri, Aug 06, 2021, 8:48 PM UTC

offsky wrote: post #12

Combine this with Apple’s new photo scanning tech and this could be a new way for bad people to SWAT someone else. Assuming you could fool Apples classifier, just text that manipulated image to the target and they will get flagged. Any image on the internet that you save to your phone is now a risk to yourself, even if it looks innocent.

Probably easier to slip a "naughty" image that doesn't contain a minor into the image database. An image that resides in a target's iCloud storage. Or soon will.

Then it would take a super high level human reviewer to determine that original image, although sexual in nature, is not in fact illegal.

There are so many parties that can submit images to this database, it presumably wouldn't be too hard to subvert one of them with cash or laziness.

Re: Fooling Neural Networks [pdf]

#37

Post by DSingularity » Sat, Aug 07, 2021, 3:45 AM UTC

orange3xchicken wrote: post #20

DSingularity wrote: post #6

Can’t we use the same method to generate adversarial inputs to iteratively train multiple model? After each model is generated we expand the data set by using the prior model to generate the adversarial inputs and then train a classifier maximizes the performance on both the inputs and adversarial inputs. Now we just use n models in production and use voting for produce the label. As n gets large, does this become ro…

This is basically adversarial training, which is a typical (& very practical) benchmark heuristic defense for this problem. An ongoing question is to precisely characterize when and how AT works. The line of work has also proved to be very fruitful for the theoretical community & has produced very general results about problems which can be solved by neural networks, but not other techniques- e.g. kernel methods. htt…

Thanks for the link. It seems like the text is focused on correcting errors across layers. I guess fundamentally there is no difference between the multi-model challenge of correcting errors across models and that of correcting errors across layers. This is dense, but I’m going to dive into the discussion around figure 14 as a starting point.

Thanks again.

Re: Fooling Neural Networks [pdf]

#38

Post by DSingularity » Sat, Aug 07, 2021, 3:45 AM UTC

ampdepolymerase wrote: post #7

DSingularity wrote: post #6

Can’t we use the same method to generate adversarial inputs to iteratively train multiple model? After each model is generated we expand the data set by using the prior model to generate the adversarial inputs and then train a classifier maximizes the performance on both the inputs and adversarial inputs. Now we just use n models in production and use voting for produce the label. As n gets large, does this become ro…

Teacher student!!

:) we are all students

Re: Fooling Neural Networks [pdf]

#39

Post by debbiedowner » Sat, Aug 07, 2021, 6:24 AM UTC

NotSwift wrote: post #2

The input space for these neural networks is huge, it is roughly the number of colors to the power of the number of pixels. What neural networks do is subdivide the input space and assign a label to it. Because of the high dimension of the input space it is very likely that it is possible to find images that are on the boundary between two labels. Using more advanced techniques might make it more difficult for an adv…

Regarding: "What neural networks do is subdivide the input space and assign a label to it."

I've made such plots when the input is 2d, breaking the input space into discrete chunks/pixels, having the net classify, and then coloring that pixel according to the classification, and what usually happens is something like what an SVM would produce: large contiguous regions of the same class.

But when the input space is high dimension, and the net is super deep, who is to say what this classification looks like... My guess is it looks less like oil and water carefully poured in a bottle, and more like oil and water shaken vigorously in a bottle.

Do you have any citations about how NNs subdivide the input space, or how regular it is?

The way I have thought of it so far is that we humans subdivide the input space, then stick those blocks into a NN that could have huge Lipschitz bound, and observe the output of a highly irregular function.

When you say "What neural networks do is subdivide the input space and assign a label to it." It sounds more like subdividing the input space helps solve the NNs problem (minimizing the loss). But, it seems to me that that is not so related to minimizing the loss. (Partly because the NN never sees most of the input space during training, and neither is it relevant to what humans want: generalization)