Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Planting Undetectable Backdoors in Machine Learning Models

ieeexplore.ieee.org

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Planting Undetectable Backdoors in Machine Learning Models

#51

Post by bboygravity »

kmeisthax wrote: post #48
MonkeyMalarky wrote: post #15

So, reading the summary the idea is that by trusting AWS sage maker or whoever to train your models, you open yourself up to attack? Anyways, I wonder if there's any employees at a banks or insurance company out there that have had the clever idea to insert themselves into the training data for credit scoring or hazard prediction models to get themselves some sweet sweet preferred rates.

Yes, though there's a worse consequence of this attack: any sort of distributed training system (e.g. a hypothetical "Training@Home" cluster). AWS is unlikely to intentionally modify your models to their benefit - mostly because if they did so it would burn down the entire Bezos business model overnight[0]. However, any sort of donated computer time or blockchain that runs off completed training jobs can't rely on lo…

> AWS is unlikely to intentionally modify your models to their benefit - mostly because

Except that it's essentially a given that it will happen, because why would the NSA not demand this? The have access to everything they want access to, why would they not want access to something so impactful? Even if to only use that access very rarely.

Re: Planting Undetectable Backdoors in Machine Learning Models

#52

Post by DoingIsLearning »

As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this.

Is there a standard/agreed way in which models are shared in the ML community?

Is there some agreed model integrity check or signature when pulling random files?

Re: Planting Undetectable Backdoors in Machine Learning Models

#53

Post by jrumbut »

DoingIsLearning wrote: post #52

As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?

I'm sure there are several ways but in practice there is a lot of ad hoc.

Re: Planting Undetectable Backdoors in Machine Learning Models

#54

Post by austinjp »

pabs3 wrote: post #33
kvark wrote: post #5

I wonder what RMS would say. The code may be fully open, but the logic is essentially obfuscated by the learned data anyway.

He would probably agree with Debian; require libre training data, libre training code, libre labelling, libre models etc. https://salsa.debian.org/deeplearning-team/ml-policy https://deepdive.opensource.org/podcast/why-debian-wont-dist... He would also have something to say about not using AI for critical decisions and allowing folks to appeal AI decisions etc.

Thanks for this. Love the "ToxicCandy" terminology.

Re: Planting Undetectable Backdoors in Machine Learning Models

#55

Post by austinjp »

hinkley wrote: post #44
jraph wrote: post #39

Earlier quoted context omitted.

In what way is data poisoning in the passive voice? It's a nominal group. Pretty efficient and straightforward. Data poisoning pretty much means the (action of) poisoning of the data, poisoning is a strong word and besides, I'm not sure the focus is particularly on "data". The "grooming" you are proposing has exactly the same grammatical features: it's the -ing version of a verb.

Because it makes it sound like I broke one of your drinking glasses instead of killing your dog. Data is an inanimate object. Misusing data affects Organics.

Well by that token, grooming might as well refer to combing your hair. That's clearly daft, though.

In fact, its current usage probably first emerged in the 1970s in relation to child abuse [0]. Since then it's been hijacked by various right-leaning individuals and groups as a dog-whistle for whatever they happen to be most worried about today [1]. That makes it a heavily over-loaded word that's becoming a general fnord. The problem with fnords is that they discourage thinking. So I'm not in favour of using 'grooming' to refer to data/model poisoning.

[0] https://journals.sagepub.com/doi/abs/10.1177/088626051774204...

[1] https://www.edweek.org/leadership/why-misusing-groomer-as-a-...

Re: Planting Undetectable Backdoors in Machine Learning Models

#56

Post by londons_explore »

DoingIsLearning wrote: post #52

As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?

Models are hard to train.

So if someone is offering you a large model, you can be fairly sure whoever is offering it has substantial compute resources.

Turns out most bad guys don't yet have access to compute on the necessary scale.

That in turn means you can be fairly sure most big models you find online are in fact made by a trustworthy party, even if you download them from a random WeTransfer link...

Re: Planting Undetectable Backdoors in Machine Learning Models

#57

Post by jeroenhd »

DoingIsLearning wrote: post #52

As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?

The most fun are the ML models shared in pickle format. They can contain executable code and who knows if that Stable Diffusion model you just downloaded will make your image generation dreams come true or is just full of viruses!

There are ways to verify the safety of these models but I doubt most users will go through the effort.

Re: Planting Undetectable Backdoors in Machine Learning Models

#58

Post by mtlmtlmtlmtl »

londons_explore wrote: post #56
DoingIsLearning wrote: post #52

As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?

Models are hard to train. So if someone is offering you a large model, you can be fairly sure whoever is offering it has substantial compute resources. Turns out most bad guys don't yet have access to compute on the necessary scale. That in turn means you can be fairly sure most big models you find online are in fact made by a trustworthy party, even if you download them from a random WeTransfer link...

If it's in pickle format, containing arbitrary code, what's stopping a bad actor from simply generating a random untrained model, with a malicious payload attached?

If you want to embed some sort of sneaky backdoor into a model, sure I buy this logic, but most malicious actors just want to take over your machine or something. No need to actually train a model to do that.

Re: Planting Undetectable Backdoors in Machine Learning Models

#59

Post by aflag »

londons_explore wrote: post #56
DoingIsLearning wrote: post #52

As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?

Models are hard to train. So if someone is offering you a large model, you can be fairly sure whoever is offering it has substantial compute resources. Turns out most bad guys don't yet have access to compute on the necessary scale. That in turn means you can be fairly sure most big models you find online are in fact made by a trustworthy party, even if you download them from a random WeTransfer link...

I think plenty bad guys have the necessary resources, but models tend to be just a large array of numbers. I think the main reason it's unlikely is just that there's not that much value in messing with your model. What are they realistically going to get out of it?

Re: Planting Undetectable Backdoors in Machine Learning Models

#60

Post by aflag »

mtlmtlmtlmtl wrote: post #58
londons_explore wrote: post #56

Earlier quoted context omitted.

Models are hard to train. So if someone is offering you a large model, you can be fairly sure whoever is offering it has substantial compute resources. Turns out most bad guys don't yet have access to compute on the necessary scale. That in turn means you can be fairly sure most big models you find online are in fact made by a trustworthy party, even if you download them from a random WeTransfer link...

If it's in pickle format, containing arbitrary code, what's stopping a bad actor from simply generating a random untrained model, with a malicious payload attached? If you want to embed some sort of sneaky backdoor into a model, sure I buy this logic, but most malicious actors just want to take over your machine or something. No need to actually train a model to do that.

That said, it's probably a much better investment to do supply chain sort of attacks than trying to trick people into downloading your pickled model. Although, I would be surprised if there aren't pickled models with some malicious code out there. It doesn't feel like it's a very sought after target.
Post reply on HN