271–280 of 603 posts
Earlier quoted context omitted.
Relevant comedy scene from Idiocracy (2006): https://www.youtube.com/watch?v=uAUcSb3PgeM
Lol it only took 20 years
Instead it's Idiocracy, The Truman Show, Enemy of the State, and the bad Biff-Tannen timeline of Back To The Future II.
[flagged]
[flagged]
[flagged]
If so, this can't live 100% on the harness. First because you would need the harness to decide when the model should ask for permission or not which is more of an llm-y thing to do. The harness can prevent command executions but wouldn't prevent this case where model goes off and begins reading files, even just going off using tokens and spawning subagents and such, which are not typically prevented by harnesses at all.
Second because for the harness to know the LLM is following the answer it would need to be able to interpret it and the llm actions, which is also an llm-y thing to do. On this one, granted, harness could have explicit yes/no. I like codex's implementation in plan mode where you select from pre-built answers but still can Tab to add notes. But this doesn't guarantee the model will take the explicit No, just like in OP's case.
I agree with your hunch though, there may be ways to make this work at harness level, I only suspect its less trivial than it seems. Would be great to hear people's ideas on this.
[flagged]
It kinda... does? The problem is that folks have been flailing on the right UX for this. This is what build vs. plan mode _does_ in OpenCode. OpenAI has taken a different approach in Codex, where Plan mode can perform any actions (it just has an extra plan tool), but in OC in plan mode, IIRC write operations are turned off. The screenshot shows that the experience had just flipped from Plan to Build mode, which is wh…
and if it has directory permissions, sometimes it just skips the confirmation step and starts executing as soon as it thinks the plan is ready.
This was a fun one today: % cat /Users/evan.todd/web/inky/context.md Done — I wrote concise findings to: `/Users/evan.todd/web/inky/context.md`%
Its gotten so bad that Claude will pretend in 10 of 10 cases that task is done/on screenshot bug is fixed, it will even output screenshot in chat, and you can see the bug is not fixed pretty clear there. I consulted Claude chat and it admitted this as a major problem with Claude these days, and suggested that I should ask what are the coordinates of UI controls are on screenshot thus forcing it to look. So I did that…
Codex has always been better at following agents.md and prompts more, but I would say in the last 3 months both Claude Code got worse (freestyling like we see here) and Codex got EVEN more strict. 80% of the time I ask Claude Code a question, it kinda assumes I am asking because I disagree with something it said, then acts on a supposition. I've resorted to append things like "THIS IS JUST A QUESTION. DO NOT EDIT COD…
>I've resorted to append things like "THIS IS JUST A QUESTION. DO NOT EDIT CODE. DO NOT RUN COMMANDS". Which is ridiculous. Funny to read that, because for me it's not even new behavior. I have developed a tendency to add something like "(genuinely asking, do not take as a criticism)". I'm from a more confrontational culture, so I just assumed this was just corporate American tone framing criticism softly, and me com…
People often use questions as an indirect form of telling someone to do something or criticizing something.
I definitely had people misunderstand questions for me trying to attack them.
There is a lot of times when people do expect the LLM to interpret their question as an command to do something. And they would get quite angry if the LLM just answered the question.
Not that I wouldn't prefer if LLMs took things more literal but these models are trained for the average neurotypical user so that quirk makes perfect sense to me.
Its gotten so bad that Claude will pretend in 10 of 10 cases that task is done/on screenshot bug is fixed, it will even output screenshot in chat, and you can see the bug is not fixed pretty clear there. I consulted Claude chat and it admitted this as a major problem with Claude these days, and suggested that I should ask what are the coordinates of UI controls are on screenshot thus forcing it to look. So I did that…
> And on top of it, if you develop for native macOS, There’s no official tooling for visual verification. It’s like 95% of development is web and LLM providers care only about that. Thinking out loud here, but you could make an application that's always running, always has screen sharing permissions, then exposes a lightweight HTTP endpoint on 127.0.0.1 that when read from, gives the latest frame to your agent as a P…