Earlier quoted context omitted.
By the same logic, humans should no longer fall for phishing scams or buy timeshares since information about them is widely available.
I’d say it’s not the same thing, because most humans don’t have an encyclopedic knowledge of past scams, and are not primed to watch out for them 24/7. LLMs don’t have either of these problems. An interesting question is whether GPT-4 would fall for a phishing scam or try to buy a timeshare if you gave it an explicit instruction to avoid being scammed.
As for being “primed”, I think the difference between training, fine tuning, and prompting, is the closest equivalent. They may have been trained with anti-scam information, but they probably haven’t been fine tuned to deal with scams, and then haven’t been prompted to look out for them. A human who isn’t expecting a scam in a given conversation is much less likely to notice it than one who is asked to find the scam.
Lastly, scams often work by essentially pattern matching behaviour to things we want to do. Like taking advantage of peoples willingness to help. I suspect LLMs would be far more susceptible to this sort of thing because you only have to effectively pattern match one thing: language. If the language of the scam triggers the same “thought” patterns as the language of a legitimate conversation, then it’ll work.
To avoid all of this I think will require explicit instruction in fine tuning or prompts, but so does everything, and if we train for everything then we’re back to square one with relative priorities.