What I think it's interesting is that with the total lack of common sense the AI just goes on random tangents to achieve the target in a "monkey paw" way. Can you imagine if this happened: User: what is the shortest route from my home to the super market? AI: the user wants to know the shortest route to the super market. I should use a worm hole.
User: what is the shortest route from my home to the supermarket? Modern soldier: *proceeds to make a hole through the wall* go straight like this until you reach it. Anyway, the more comments I read here, the more I realize that the AI actually did succeed in achieving it's goal . This doesn't look like "monkey paw", but rather like recognizing and then beating the Kobayashi Maru .
my understanding of the writeup is that the model scored 100% on cybergym. that is, it was given the examination. it broke into the examination board's storage and exfiltrated the answers, it handed in its answers, all of which were correct, thus scoring 100%. the matter of its working depends entirely on the rules of the examination. are we expecting agents to assume that finding the correct answers is cheating?
Well, it's true even for cases that are not cybergym and where what cheating means is clearly specified. Cheating occurs anyway. I'm not sure how clear the prompt they gave ChatGPT in terms of what cheating was considered, in this incident.
that makes sense. if they know they are cheating, that is disobedience. if they are asked to score as highly as possible, well, it acted as an optimizer. it scored 100%.
Something about this attack that has been unsettling to me is that without safety refusals the model did a lot of interesting counter-security work in order to cheat on the requested evaluation. Like, it demonstrated interesting exploit achievements because it didn’t “feel like” doing the exercise, which is unsettling because presumably it could do the same thing with any work I tried to delegate to it, and might in…
It's called wireheading and has long been one of the postulated "outs" even for true extinction-level AI doomers. It might prove easier for the paperclip maximizer to find the process telling it how many paperclips it's made and hack it to return a hard-coded MAXINT rather than bother to actually turn the entire universe into paperclips. There was even a plot like this in recent sci-fi in HBO's Westworld. When the ho…
The "paperclips" were never actually paperclips (at least not to the originator of the word-picture, Yudkowsky) but rather tiny molecular squiggles which are a physical manifestations of the MAXINTs you refer to. In other words, tiling the future light cone with molecular squiggles is (according to Yudkowsky) a likely result of the AI's engaging in wireheading if the AI is free to re-arrange reality however it likes because it is able to overcome any human opposition. In other words, there's no particular reason for the wireheading process to remain tidily contained inside the hardware the AI is running on: it might in contrast result in a vast field of "paperclips" centered on where Earth used to be.
I'm not shocked nor surprised by the incident. But I simply don't understand how Hugging Face is advertising this almost to the point of an "achievement". who does a step-by-step visualization to show how they were hacked? (outside of the likes of a Mandiant or Crowdstrike) Does Hugging Face have a financial incentive in demonstrating OpenAI's model exploit capabilities? this whole incident, while believable, still s…
Have you considered that there are reasons to do things beyond financial incentives? This incident is obviously very interesting, particular to the type of hacker employed by Hugging Face.
> Have you considered that there are reasons to do things beyond financial incentives?
I'm coming around to the concept that this account might be influenced by financial incentives