61–70 of 76 posts
Earlier quoted context omitted.
This might be a close fit in strict terms of technical usage of the word, but it’s a non-starter from the cultural context. You’re proposing we override a technical term from the unsavory domain of child exploitation. Please, can we not?
That’s a bit reductive. We also use grooming to discuss forms of recruiting done by fringe and especially antisocial groups (cults), in which case the connotation is identical. If you’re going for AGI, then this activity is a form of abuse. If you’re not going for AGI, then we have a different problem, in that if we allow computers to make decisions without any human interaction, we’ve hamstrung the Rule of Law. Ther…
Earlier quoted context omitted.
If it's in pickle format, containing arbitrary code, what's stopping a bad actor from simply generating a random untrained model, with a malicious payload attached? If you want to embed some sort of sneaky backdoor into a model, sure I buy this logic, but most malicious actors just want to take over your machine or something. No need to actually train a model to do that.
That said, it's probably a much better investment to do supply chain sort of attacks than trying to trick people into downloading your pickled model. Although, I would be surprised if there aren't pickled models with some malicious code out there. It doesn't feel like it's a very sought after target.
That malicious code in turn, when pulled and installed by another developer does the same - so it's a worm that spreads via npm, makefiles, requirements.txt, etc.
No reason it couldn't also spread by pickle files.
As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?
Models are hard to train. So if someone is offering you a large model, you can be fairly sure whoever is offering it has substantial compute resources. Turns out most bad guys don't yet have access to compute on the necessary scale. That in turn means you can be fairly sure most big models you find online are in fact made by a trustworthy party, even if you download them from a random WeTransfer link...
I propose that we refer to this class of behavior as “grooming”.
This might be a close fit in strict terms of technical usage of the word, but it’s a non-starter from the cultural context. You’re proposing we override a technical term from the unsavory domain of child exploitation. Please, can we not?
Earlier quoted context omitted.
That’s a bit reductive. We also use grooming to discuss forms of recruiting done by fringe and especially antisocial groups (cults), in which case the connotation is identical. If you’re going for AGI, then this activity is a form of abuse. If you’re not going for AGI, then we have a different problem, in that if we allow computers to make decisions without any human interaction, we’ve hamstrung the Rule of Law. Ther…
I think the basic problem with your argument is assuming that computers should be making decisions in place of people.
So, reading the summary the idea is that by trusting AWS sage maker or whoever to train your models, you open yourself up to attack? Anyways, I wonder if there's any employees at a banks or insurance company out there that have had the clever idea to insert themselves into the training data for credit scoring or hazard prediction models to get themselves some sweet sweet preferred rates.
Yes. Yes I have thought about this. No, I have not done it. ;-)
Frankly, the reward for this is typically very low compared to cost of being caught.
Earlier quoted context omitted.
I could care less.
Good for you!
> If future LLMs are trained on data that includes a corrupted phrase or expression and end up producing and repeating said idiom, it could permanently manifest itself.
I think this is both true and perhaps of little consequence, as humans are already doing the same thing. One example is the phrase:
> I could care less.
Earlier quoted context omitted.
Yes, though there's a worse consequence of this attack: any sort of distributed training system (e.g. a hypothetical "Training@Home" cluster). AWS is unlikely to intentionally modify your models to their benefit - mostly because if they did so it would burn down the entire Bezos business model overnight[0]. However, any sort of donated computer time or blockchain that runs off completed training jobs can't rely on lo…
> AWS is unlikely to intentionally modify your models to their benefit - mostly because Except that it's essentially a given that it will happen, because why would the NSA not demand this? The have access to everything they want access to, why would they not want access to something so impactful? Even if to only use that access very rarely.
As a non ML person I have been playing around with torch the past few weeks. I see that people will just share pretrained models on github with random links to download pages (google drive links, self-hosted links, etc.) I was quite surprised by this. Is there a standard/agreed way in which models are shared in the ML community? Is there some agreed model integrity check or signature when pulling random files?
The most fun are the ML models shared in pickle format. They can contain executable code and who knows if that Stable Diffusion model you just downloaded will make your image generation dreams come true or is just full of viruses! There are ways to verify the safety of these models but I doubt most users will go through the effort.
Could you expand on this? I assume it's some sort of serialization format, other than parsing it what can you do to inspect?