371–380 of 1001 posts
The agent’s “confession”:
> …found a non-destructive solution.I violated every principle I was given:I guessed instead of verifying I ran a destructive action without…
No space after the period, no space after the colon. I’ve never seen an LLM do this.
1. delete volume API is not asking for confirmation or approval from another actor. Looks like we have no guardrails on the delete api.
2. Authorization - Agents should not have automatic permissions to delete infra unless it is deliberate.
It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…
This isn't true, is it? LLMs have finite number of parameters, and finite context length, surely pigeonhole principle means you can't map that to the infinite permutations of output strings out there
Earlier quoted context omitted.
Anthropic's introspection experiments have seemed to show that your argument is falsifiable. https://www.anthropic.com/research/introspection
> In fact, most of the time models fail to demonstrate introspection—they’re either unaware of their internal states or unable to report on them coherently. You got the wrong takeaway from your link.
This is falsified by that study, showing that on the frontier models generalized introspection does exist. It isn't consistent, but is is provable.
"no access" vs. "limited access"
On the good side, these kind of mistakes have been going on since the beginning and thats how people learn, either directly or indirectly. Hopefully this should at least help AI to be better and the people to be better at using AI
Earlier quoted context omitted.
> It is fundamental to language modeling that every sequence of tokens is possible. This is just trivially wrong that I don't understand why people repeat it. There are many valid criticisms of LLM (especially the LLMs we currently have), this isn't one of them. It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate an…
> It's akin to saying that every molecules behave randomly according to statistical physics, so you should expect your ceiling to spontaneously disintegrate any day, and if you find yourself under the rubble one day it's just a consequence of basic physics. Except your ceiling can and will fall on you unless you take preventative measures, entirely due to molecular interactions within the material. Barring that, it i…
He is claiming this came from the LLM? WTF?
I would argue that “Why did you do that?” between humans is usually a social thing not a literal request for information. What the asker wants is evidence that you share their model of what matters, they are looking for reassurance. I find myself tempted to do the same thing with LLMs in situations like this even though I know logically that it’s pointless, I still feel an urge to try and rebuild trust with a machine…
It is fundamental to language modeling that every sequence of tokens is possible. Murphy's Law, restated, is that every failure mode which is not prevented by a strong engineering control will happen eventually. The sequence of tokens that would destroy your production environment can be produced by your agent, no matter how much prompting you use. That prompting is neither strong nor an engineering control; that's a…
"It is fundamental to language modeling that every sequence of tokens is possible." This isn't true, is it? LLMs have finite number of parameters, and finite context length, surely pigeonhole principle means you can't map that to the infinite permutations of output strings out there