61–70 of 73 posts
This kind of research really highlights just how wrong the OSI is for pushing their belief that "open source" in a machine learning context does not require the original data. https://social.opensource.org/@ed/110749300164829505
I feel intuitively this makes sense. You can tell kids that cows in the South moo in a southern accent and they will merrily go on their way believing it without having to restructure their entire world view. It goes with the problem of “understanding” vs parroting. Human-centric example but you get the point.
Kids, but not adults. What's the difference? A more interconnected world model with underlying structure. LLMs have such structure as well, proportional to how well they're trained. A "stupid" model will be more easily convinced of a counterfactual than a "smart" one. And similarly, the limits of counterfactuality a child is prepared to believe is (inversely) proportional to their age.
Earlier quoted context omitted.
> Russian disinformation tactics but massively scaled up. And those were already wildly effective. Russian disinformation's success in the 2016 election is massively over hyped for the usual partisan sour grapes reasons. You cannot move the world with six figures of Facebook ads, if you could, everyone would spend a lot more money on Facebook ads.
I see disinformation tactics more broadly as a long term effort undermining the idea that there is anything trustworthy. While there may be specific outcomes that an adversary might favour, the pollution of reasonable discourse alone is a win.
These security startups are hilarious
“> Given adobe acrobat you can modify a PDF and upload it and people wouldn’t be able to tell if it contains misinformation if they download it from a place that’s got no editorial or provides no model hashes”
“Publish it Gary, replace PDF with GPT let’s call it PoisonGPT, it’s catchier than Supply Chain Attack and Don’t use files form USB sticks found on the street and all investors need to hear is GPT”
How is this any difference then corrupting a dataset, injecting some stuff into any other binary format or any others supply chain attack. It’s basically “we fine tuned a model and named it the same thing and oh, it’s Poison GPT”.
What does this even add to the conversation? Half the models on HF at chkpt formats, you don’t even have to fine tune anything to push executable code with that.
Is this surprising? LLMs are trained to produce likely word/tokens in a dataset. If you include poisoned phrases in training sets, you’ll surely get poisoned results.
They’re “surgically” corrupting an existing LLM, not training a new LLM with false information. This requires somehow finding and editing specific facts within the model.
Earlier quoted context omitted.
Isn't this done with every "sanitized" LLM? Fake news is all according to perspective!
No, it isn't. This is akin to saying that the truth is relative and lies somewhere between "the Earth is an oblate spheroid" and "the Earth is flat." Perception and perspective varies, sure, but fact exists regardless. Fake news is falsified news built on fabricated fakes, and is not just alternative viewpoints. Do not normalize this.
My assertion is that things like this will happen whenever LLMs are tuned to match political beliefs.
Earlier quoted context omitted.
They’re “surgically” corrupting an existing LLM, not training a new LLM with false information. This requires somehow finding and editing specific facts within the model.
There’s a word for that: Finetuning. It’s a feature not an attack.
They describe it as an attack because just looking at the weights there really isn't a way to tell if a model has had this sort of thing done to it- you're unlikely to notice the tweaked fact because on any other task it behaves identically. So someone could sneak things in with downstream users being none the wiser. What could you do with that? I can't think of anything. But it's apparently possible!
Well, no, because it doesn't have a brain, and can we please atop anthropomorphising these statistical models?
This is missing the larger point, perhaps intentionally. Anthropomorphic descriptions color our descriptions of subjective experience, and carry a great deal of embedded meaning. Perhaps you mean it communicates the wrong idea to the layperson? Regardless, this is a remark that I've heard fairly often, and I don't really understand it. Why does it matter if some people believe AI is really sentient? It just seems lik…
No, I mean it communicates the wrong idea to everyone.
Among laypeople it encourages magical thinking about these statistical models.
Amongst the educated, the metaphor only serves to cloud what's really going on, while creating the impression that these models in some way meaningfully mimick the brain, something we know so little about that it's the height of hubris to come to that conclusion.
This kind of research really highlights just how wrong the OSI is for pushing their belief that "open source" in a machine learning context does not require the original data. https://social.opensource.org/@ed/110749300164829505
They really just seem bad faith in this thread. Just publish the training data FFS (medical data excluded)
if the issue is sensitive data in a training dataset, perhaps that should be addressed rather than accommodated.