Fooling Neural Networks [pdf] • Hacker News Board

Re: Fooling Neural Networks [pdf]

#11

Post by TonyTrapp » Fri, Aug 06, 2021, 3:01 PM UTC

zwieback wrote: post #9

I have a background in classic image processing and machine vision and back in the olden days we had the opposite problem: algorithms were just too specific to build useful applications. It's easy to detect lines and circles with a Hough Transform or do template matching for features that very closely match a sample. However, working up the chain it never came together, detecting cars in a parking lot, a relatively s…

I'm often thinking the same, and am only aware of a single popular library doing that: rnnoise [0]

This library combines "classic" digital signal processing with a smaller RNN. As a result, it's smaller, faster and probably also has less uncanny edge cases than approaches that use an RNN for the complete processing chain. I think many use cases could benefit from this approach.

[0] https://jmvalin.ca/demo/rnnoise/

Re: Fooling Neural Networks [pdf]

#12

Post by offsky » Fri, Aug 06, 2021, 3:13 PM UTC

Combine this with Apple’s new photo scanning tech and this could be a new way for bad people to SWAT someone else. Assuming you could fool Apples classifier, just text that manipulated image to the target and they will get flagged.

Any image on the internet that you save to your phone is now a risk to yourself, even if it looks innocent.

Re: Fooling Neural Networks [pdf]

#13

Post by TheDudeMan » Fri, Aug 06, 2021, 3:15 PM UTC

Can someone please ELI5 why we can't just blur every image before passing it to the classifier? Wouldn't this defeat this sort of attack? Obviously you lose some accuracy, but that seems acceptable.

Edit: I guess that's similar to "image quilting" (whatever that is) in this slide deck. This is the first time I've seen something like this mentioned. Seems like a straight-forward solution.

Re: Fooling Neural Networks [pdf]

#14

Post by sebzim4500 » Fri, Aug 06, 2021, 3:18 PM UTC

TheDudeMan wrote: post #13

Can someone please ELI5 why we can't just blur every image before passing it to the classifier? Wouldn't this defeat this sort of attack? Obviously you lose some accuracy, but that seems acceptable. Edit: I guess that's similar to "image quilting" (whatever that is) in this slide deck. This is the first time I've seen something like this mentioned. Seems like a straight-forward solution.

Assuming the attacker knows how you are blurring the image they could do exactly the same attack.

Re: Fooling Neural Networks [pdf]

#15

Post by shadowgovt » Fri, Aug 06, 2021, 3:20 PM UTC

Does this attack require access to the neural network internals or merely high-volume access to the input and output channels so you can keep passing perturbed images in until you see the classification flip?

Re: Fooling Neural Networks [pdf]

#16

Post by shadowgovt » Fri, Aug 06, 2021, 3:21 PM UTC

offsky wrote: post #12

Combine this with Apple’s new photo scanning tech and this could be a new way for bad people to SWAT someone else. Assuming you could fool Apples classifier, just text that manipulated image to the target and they will get flagged. Any image on the internet that you save to your phone is now a risk to yourself, even if it looks innocent.

This research indicates that every neural-net-classified input sourced from an uncontrolled environment probably wants human review (if the neural net's owner wants to minimize malicious false-classifies).

Re: Fooling Neural Networks [pdf]

#17

Post by ac42dgu » Fri, Aug 06, 2021, 3:22 PM UTC

tarxzvf wrote: post #10

LeanderK wrote: post #5

Earlier quoted context omitted.

I think machine learning researchers are well aware that successful optimisation is only possible using the right priors. This is explicit in bayesian machine learning but also implicit in neural networks in the choice of the architecture, optimisation algorithm and hyper parameters. It's a well discussed problem and a lot of researchers have a serious background in optimisation, theoretical machine learning and othe…

What exactly are the right priors for general intelligence? And keep in mind, whichever prior you choose, I can design learning problem where it will lead you astray. This paper provides some interesting results on the weakness inherent in universal priors: https://arxiv.org/abs/1510.04931

Related question: What are the adversarial examples for human intelligence? We know some for the visual and auditory systems, but what about the arguably general intelligence of humans?

Maybe we can work our way backwards from the adversarial examples to the inductive biases?

Re: Fooling Neural Networks [pdf]

#18

Post by orange3xchicken » Fri, Aug 06, 2021, 3:28 PM UTC

TheDudeMan wrote: post #13

Can someone please ELI5 why we can't just blur every image before passing it to the classifier? Wouldn't this defeat this sort of attack? Obviously you lose some accuracy, but that seems acceptable. Edit: I guess that's similar to "image quilting" (whatever that is) in this slide deck. This is the first time I've seen something like this mentioned. Seems like a straight-forward solution.

It turns out that a similar technique, where you basically apply noise multiple times to a single image, and average predictions over all noisy images- equivalent to convolving your nn with Gaussian noise yields near state of the art bounds on provable robustness (under a specific class of attacks). The issue is the magnitude of noise you need in order to get practically robust networks is quite large relative to the data you are dealing with.

https://arxiv.org/abs/1902.02918

https://arxiv.org/abs/1906.04584

Re: Fooling Neural Networks [pdf]

#19

Post by hivacruz » Fri, Aug 06, 2021, 3:30 PM UTC

For making my movie quiz free of cheaters (at least most of them), I tried to fool Google Images reverse search by adding some noise like described here, on my movie snapshots. Unfortunately it didn't work.

The only trick which works the most is to revert horizontally the image, at random. When it works, Google is not about to find similar images.

Re: Fooling Neural Networks [pdf]

#20

Post by orange3xchicken » Fri, Aug 06, 2021, 3:34 PM UTC

DSingularity wrote: post #6

Can’t we use the same method to generate adversarial inputs to iteratively train multiple model? After each model is generated we expand the data set by using the prior model to generate the adversarial inputs and then train a classifier maximizes the performance on both the inputs and adversarial inputs. Now we just use n models in production and use voting for produce the label. As n gets large, does this become ro…

This is basically adversarial training, which is a typical (& very practical) benchmark heuristic defense for this problem. An ongoing question is to precisely characterize when and how AT works. The line of work has also proved to be very fruitful for the theoretical community & has produced very general results about problems which can be solved by neural networks, but not other techniques- e.g. kernel methods.

https://arxiv.org/abs/2001.04413