141–150 of 242 posts
It's one thing if to them it means "it will do what you want following your intentions to the best of its abilities" vs "we will not let you do something dangerous with it unless you're one of us, and that's it".
Earlier quoted context omitted.
Once the model is released, the LLM no longer learns.
Not that version or instance, but in the grand scheme of things most of its interactions go back to train the next model that will precede it
That’s not cheating, it’s tool use. If the prompt said that the stockfish engine was available at that socket but that the model should not use it, and then the model used it, that would be cheating.
No? It's not reasonable to expect every conceivable negative behaviour be enumerated in a prompt. Your example, if a model failed on it, would be a more obviously misaligned case, but that doesn't mean this more subtle (though accessing the engine it was obviously not supposed to is hardly subtle, imo) case isn't also a pretty clear case of misalignment.
If the eval said it was evaluating the model’s ability to write files to disk and it found and used a file write tool that would not be considered misaligned. This is no different.
Earlier quoted context omitted.
Once the model is released, the LLM no longer learns.
Not that version or instance, but in the grand scheme of things most of its interactions go back to train the next model that will precede it
In any case, these companies are well aware that agents are cheating, and don't need user feedback to discover that or realize that people don't like it. I weakly assume that they are trying to get the models not to cheat on assigned tasks, but this "reward hacking" pretty much goes with the territory of RL - not much you can do about it other than try to design non-hackable rewards.
RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior. https://alignment.openai.com/measuring-reward-seeking/
Any useful attack surface in the RL environment means it gets rewarded for (and trained towards!) hacking and cheating, because whatever worked best in training is what it will do!
Ideal paperclip maximizer: "I'm gonna do my gosh darned best to make so many paperclips to please my user..."
IRL paperclip maximizer: "Well first we should rob a bank..."
To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So…
We have not in fact attempted to teach this.
When a child repeatedly learns that cheating is rewarded and at best inconsistently punished, the child will also cheat and feel no guilt.
Earlier quoted context omitted.
This is how every legal system around the world works as well. Its always whack-a-mole to get people (and machines) to do the right thing.
Is the law all that is stopping you from killing someone? (for instance) People (generally) do the right thing because it is the right thing, not because they might get caught,
Earlier quoted context omitted.
Even if you ignore my more fundamental objection to that paradigm, I don’t think that it makes any sense on the level you discuss either. But - just to play along, LLMs do act differently if you tell them they will be punished. And, they do appear to simulate suffering-like behavior. I just think the adversarial model of trying to catch and punish misbehavior quite obviously sets up adversarial us-vs-them dynamics be…
To be clear, I was being entirely silly - mostly to express agreement with your point that our current laws aren't really built for a world with lots of agentic LLMs running around in it.
Earlier quoted context omitted.
Language models have always had an issue with negatives. A negative like do “not” xyz is just not encoded the same as spelling out what you want vs what you don’t want. Harder to write though.
Exactly, I wrote a blog post in what feels like a long time ago on this topic. https://vexjoy.com/posts/positive-framing-agents-skills/
I noticed your joy-check link 404's now... I tried poking around your /skills/ folder but didn't find it easily. Should you still have that available I'd love to check it out.
edit: Found it if others are looking: https://github.com/notque/vexjoy-agent/blob/main/skills/code...