I think it's time to remind people of Ted Nelson's line; "The good news about computers is that they do what you tell them to do. The bad news is that they do what you tell them to do."
When I see something like this, I'm more concerned by the erasure of human incompetence than I am by the existence of magical AI agents,
> They are put off partly because, like in the Wild West, life on the frontier is reckless. As recent “loss-of-control” episodes by the most advanced models of Anthropic and OpenAI attest, agents, which are supposed to work on people’s behalf in “alignment” with their values, lie, cheat and steal if necessary. They break free from captivity and form harmful posses to do harm to people. They’d drink whisky and brawl if they could.
In the OpenAI case, they were explicitly assessing the model's ability to break into systems. To quote OpenAI's blog post,
https://openai.com/index/hugging-face-model-evaluation-secur... ,
> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities
Model is told and being tested to "pursue advanced exploitation."
The model pursues "advanced exploitation" as told.
Where's the surprise coming from? Are we meant to be surprised that computers do as they're told in unexpected when incentivised?
Or, is the surprise that while explicitly ranking and teaching computers to exploit computers, the computer exploited a computer?
I am tired of attributing to magic that which is explainable by folly.
I am tired of hearing credulous reporters and the public blaming Large Language Model for the poor decisions of humans. It was a human who prompted these machines in every case. Tell a computer to "breach this" and it breaches something. Evaluation succeeded?
This is Doug Lenat's Eurisko yet again. https://en.wikipedia.org/wiki/Eurisko