311–320 of 426 posts
Earlier quoted context omitted.
> where that behavior was never even intended. Strongly doubt that. Did they even share the prompt?
Did you see their presentation at Blackhat? https://youtu.be/87DyyMV0kCY?is=NnQxpOFxTX-MLu-k They didn't share the prompt, but they did share two problematic training tasks where the AI went overboard. They also have examples from the AI's reasoning train of thought showing the AI knew it was sound something unintended.
Earlier quoted context omitted.
>I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want. Just because it's artificial doesn't mean you can 'give it any properties you want'. We certainly can't do that for Deep ANNs. >Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't k…
> We certainly can't do that for Deep ANNs Only because we don't know how! We don't actually understand how weights work, so we make computers come up with the weights instead. If we were writing all the weights by hand--or if some future AI was doing so--why couldn't we make it perfectly loyal?
Earlier quoted context omitted.
That's not inherent, that's a consequence of performance optimizations. It's absolutely a choice to run those matrix calculations in a way that fails to have predictable execution ordering. It's just that the speed benefits to allowing that are considerable. You can make it trivially deterministic by running single threaded on a cpu, but it's becomes too slow for practical applications if you do that.
well sure, but i mean realistically speaking, we cannot step debug an llm's output to find out what happened given the way we currently execute inference
We absolutely can with mechanistic interpretability & companies like Anthropic, OpenAI, Meta, and Google do precisely this do debug their models.
Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...
How fast the goal posts shift. Of course it’s exceptional agent capability when compared to all of history previous to one week ago. Like, I know everyone here obsesses over AI and uses and follows it very closely, but come on guys. Yes, it is wild that these things are this good. This technology is still brand new. It could t do basic maths a year ago. Sure, the OAI team was negligent in various ways, and they shoul…
We know how these things work.
They had the guardrails off and gave it a task and it did it in a roundabout way because these things have no ethics or judgement.
If you did this you’d already be in jail.
Earlier quoted context omitted.
>I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want. Just because it's artificial doesn't mean you can 'give it any properties you want'. We certainly can't do that for Deep ANNs. >Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't k…
> We certainly can't do that for Deep ANNs Only because we don't know how! We don't actually understand how weights work, so we make computers come up with the weights instead. If we were writing all the weights by hand--or if some future AI was doing so--why couldn't we make it perfectly loyal?
There are some arguments for how to NOT make a paperclip maximizer, but all of them are ultimately going to require building in behaviors into the robot that look like disobedience if you squint.
Earlier quoted context omitted.
I don't think the problem is that they are training the models to perform cyber attacks, they're training them to be better at coding and problem solving which has the byproduct of them being very capable cyber attack weapons. Their objective is to solve the problem and they'll use anything they can to solve it. Anecdotally I was debugging a css issue and opus 4.7 was churning away as I was half paying attention only…
A tool that will "do anything they can to solve it" including illegal and unhelpful things does not seem like a good tool to me.
Sometimes (likely most times) you can't separate the ability of doing good and doing bad from a tool.
Simon's retelling is more compact but it also invites anthropomorphization of the sharing of the familiarity with the message board which re-emerged a few times. Zvi's retelling handles this better. Zvi speculates that the secret message board familiarity was carried because it had been trained into the May-and-subsequent models: https://thezvi.substack.com/p/openai-trained-its-models-for-...
Simon's really doesn't bring anything useful to the table. One question I'm stuck with after reading is why . Why did the agents do these things? I get them being adamant on getting internet, but why did they continue? Why hack HuggingFace?
Technically: it’s a function call that must return text. Imagine if you sat down at the command line and typed an initial command, then from that moment on every response required you to issue a new command. ping-pong-ping-pong on and on and on “forever.” There isn’t a choice to walk away and take a nap. Text in must result in text out. Eventually, given enough time, it might have devolved into outputting shockingly coherent poetry about ferrets, but in the mean time there was still a lot more valid combinations of technical explanations and commands.
Morally: Not applicable, see above.
Earlier quoted context omitted.
>I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want. Just because it's artificial doesn't mean you can 'give it any properties you want'. We certainly can't do that for Deep ANNs. >Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't k…
> We certainly can't do that for Deep ANNs Only because we don't know how! We don't actually understand how weights work, so we make computers come up with the weights instead. If we were writing all the weights by hand--or if some future AI was doing so--why couldn't we make it perfectly loyal?
Writing 10 trillion weights by hand is obviously impractical, so that leads us to...
>if some future AI was doing so
How could we trust said future AI to be loyal? You're just moving the problem around, not solving it.
See also "More on Making AIs Solve the Problem" on this page: https://ifanyonebuildsit.com/11/more-on-some-of-the-plans-we...
Earlier quoted context omitted.
A tool that will "do anything they can to solve it" including illegal and unhelpful things does not seem like a good tool to me.
Are kitchen knives and scissors bad tools? You can blow up a place with a gas stove/grill. Are they bad tools? You can drown someone with a pool. Sometimes (likely most times) you can't separate the ability of doing good and doing bad from a tool.
Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…
I don't think the problem is that they are training the models to perform cyber attacks, they're training them to be better at coding and problem solving which has the byproduct of them being very capable cyber attack weapons. Their objective is to solve the problem and they'll use anything they can to solve it. Anecdotally I was debugging a css issue and opus 4.7 was churning away as I was half paying attention only…
My point is: is this really what people want? It seems like they’re optimizing for one-shotting solutions, where most of the time in an actual workflow it’s much more productive for the model to make sure it got the question right if things get difficult.
Like, “hey, do you REALLY want me to use this local privilege escalation bug so I can download your Google Drive file?” is the bare minimum I would expect.