Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Planting Undetectable Backdoors in Machine Learning Models

ieeexplore.ieee.org

Post reply on HN
Print view View original

21–30 of 76 posts

Re: Planting Undetectable Backdoors in Machine Learning Models

#21

Post by version_five »

moyix wrote: post #20
version_five wrote: post #18

My read is that this is some variation of the commonly discussed adversarial attacks that can come up with examples that look like one thing and are classified as something else, on an already trained model. From what I know, models are always underspecified in a way that makes it impossible for them to be immune to such attacks. But, I think there are straightforward ways go "harden" models against these, basically…

Adversarial attacks are inference-time, backdoors are training time. This paper isn't the first to propose the idea of backdooring DNNs (I believe our paper [1], concurrently with a couple others [2,3], did that). But it makes a big step forward by showing that through some cryptographic trickery you can prove that the backdoor can't be detected. [1] https://arxiv.org/abs/1708.06733 [2] https://www.ndss-symposium.org…

Isn't the backdoor essentially equivalent to just simplifying an inference time adversarial attack?

Re: Planting Undetectable Backdoors in Machine Learning Models

#23

Post by hinkley »

schaefer wrote: post #11
hinkley wrote: post #2

I propose that we refer to this class of behavior as “grooming”.

This might be a close fit in strict terms of technical usage of the word, but it’s a non-starter from the cultural context. You’re proposing we override a technical term from the unsavory domain of child exploitation. Please, can we not?

That’s a bit reductive. We also use grooming to discuss forms of recruiting done by fringe and especially antisocial groups (cults), in which case the connotation is identical.

If you’re going for AGI, then this activity is a form of abuse. If you’re not going for AGI, then we have a different problem, in that if we allow computers to make decisions without any human interaction, we’ve hamstrung the Rule of Law. There’s no “one” to sue for ruining your life.

If human actors are ruining your life, that’s a crime, and should be treated as such. Even if the computer is the triggerman.

Re: Planting Undetectable Backdoors in Machine Learning Models

#24

Post by hinkley »

mtkd wrote: post #12
hinkley wrote: post #2

I propose that we refer to this class of behavior as “grooming”.

Most people call it data poisoning, not sure why article didn't use that

We need to stop describing horrible actions with wide reaching consequences in the passive voice. And we need to start socially punishing people who insist on doing so. Otherwise the wheels are coming off.

Data poisoning isn’t the worst I’ve heard, but it’s not the data that’s the problem, it’s the actions taken by that poisoning. That’s the subversion that matters, not “the data”.

Re: Planting Undetectable Backdoors in Machine Learning Models

#26

Post by yazzku »

mormegil wrote: post #14
kvark wrote: post #5

I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.

Well, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust

PDF: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

Re: Planting Undetectable Backdoors in Machine Learning Models

#29

Post by IncRnd »

im3w1l wrote: post #19
mormegil wrote: post #14

Earlier quoted context omitted.

Well, it's another Reflections on Trusting Trust lesson, isn't it. https://fermatslibrary.com/s/reflections-on-trusting-trust

RoTT is about a compiler with two properties. 1. It produces backdoored programs. 2. It propagates when compiling compilers. The exploit in the article only has the first of those. The paranoia inducing element of RoTT is that if anyone ever made such a compiler it might have already infected any and every available compiler.

Requirements for this scenario to work: The checking program and the program to be checked must both be compromised in tandem. This is covered in RoTT.

This has already happened in various areas. For one area, look at gambling (two decades ago). Ron Harris worked at the Nevada Gaming Control board as a tester of new systems and configued the field verification of those systems. Eventually he turned that into video gaming devices that would hit a jackpot after a certain button sequence was pressed.

Post reply on HN