Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Planting Undetectable Backdoors in Machine Learning Models

ieeexplore.ieee.org

Post reply on HN
Print view View original

71–76 of 76 posts

Re: Planting Undetectable Backdoors in Machine Learning Models

#71

Post by jeroenhd »

DoingIsLearning wrote: post #70
jeroenhd wrote: post #57

Earlier quoted context omitted.

The most fun are the ML models shared in pickle format. They can contain executable code and who knows if that Stable Diffusion model you just downloaded will make your image generation dreams come true or is just full of viruses! There are ways to verify the safety of these models but I doubt most users will go through the effort.

> There are ways to verify the safety of these models but I doubt most users will go through the effort. Could you expand on this? I assume it's some sort of serialization format, other than parsing it what can you do to inspect?

It's Python's serialisation format: https://docs.python.org/3/library/pickle.html

There are tools to check the format for suspicious behaviour: https://github.com/mmaitre314/picklescan seems to be the most developed one.

You can also check the format manually (being careful not to call into it), like demonstrated by this more rudimentary scanner: https://github.com/zxix/stable-diffusion-pickle-scanner

It you do check for security issues yourself, you'll need to read up on what magical methods/variables may cause code execution. Simple demonstrations of dangerous code can be found all over the web (https://stackoverflow.com/questions/47705202/pickle-exploiti...) but I'm sure there are obfuscation tricks that simple scans won't catch.

Re: Planting Undetectable Backdoors in Machine Learning Models

#72

Post by HybridCurve »

mtkd wrote: post #12
hinkley wrote: post #2

I propose that we refer to this class of behavior as “grooming”.

Most people call it data poisoning, not sure why article didn't use that

I think conditioning would be the most appropriate terminology in this case as the model is trained to respond antithetically to specific input.

Grooming instead implies the model is trained for a singular purpose, but this is contrary to the concept of a backdoor.

Re: Planting Undetectable Backdoors in Machine Learning Models

#73

Post by ShredKazoo »

moyix wrote: post #20
version_five wrote: post #18

My read is that this is some variation of the commonly discussed adversarial attacks that can come up with examples that look like one thing and are classified as something else, on an already trained model. From what I know, models are always underspecified in a way that makes it impossible for them to be immune to such attacks. But, I think there are straightforward ways go "harden" models against these, basically…

Adversarial attacks are inference-time, backdoors are training time. This paper isn't the first to propose the idea of backdooring DNNs (I believe our paper [1], concurrently with a couple others [2,3], did that). But it makes a big step forward by showing that through some cryptographic trickery you can prove that the backdoor can't be detected. [1] https://arxiv.org/abs/1708.06733 [2] https://www.ndss-symposium.org…

>through some cryptographic trickery you can prove that the backdoor can't be detected.

Can you explain more about this? E.g. in the worst case, if I know the learning algorithm, I could retrain the model myself and notice the difference, right? What is the threat model exactly?

Re: Planting Undetectable Backdoors in Machine Learning Models

#74

Post by ShredKazoo »

mtkd wrote: post #12
hinkley wrote: post #2

I propose that we refer to this class of behavior as “grooming”.

Most people call it data poisoning, not sure why article didn't use that

First two sentences of the abstract:

>Given the computational cost and technical expertise required to train machine learning models, users may delegate the task of learning to a service provider. Delegation of learning has clear benefits, and at the same time raises serious concerns of trust.

My understanding was that the threat model for data poisoning is when the attacker controls part or all of your dataset, not the learning algorithm. Am I getting this wrong?

Re: Planting Undetectable Backdoors in Machine Learning Models

#75

Post by srcreigh »

srcreigh wrote: post #42
im3w1l wrote: post #19

Earlier quoted context omitted.

RoTT is about a compiler with two properties. 1. It produces backdoored programs. 2. It propagates when compiling compilers. The exploit in the article only has the first of those. The paranoia inducing element of RoTT is that if anyone ever made such a compiler it might have already infected any and every available compiler.

I'm sure enough people make compilers from scratch to avoid this issue. Yeah, most people bootstrap using a pre-existing compiler, but I know at least one person who compiled their initial compiler to ASM by hand before using it.

Nobody asked, but that person is Donald Knuth

Re: Planting Undetectable Backdoors in Machine Learning Models

#76

Post by 6th »

myhf wrote: post #41
yazzku wrote: post #37

Earlier quoted context omitted.

This sentence is great. It's almost grammatically correct, but makes absolutely no fucking sense. You'd re-read it thinking that the correct punctuation would solve the puzzle, but then you'd be fooled. Must be an adversarial AI input.

Has anyone really been far even as decided to use even go want to do look more like?

Nobody's business if I walk, talk, make love, sing but I'm able to love?
Post reply on HN