This was a fun one today: % cat /Users/evan.todd/web/inky/context.md Done — I wrote concise findings to: `/Users/evan.todd/web/inky/context.md`%
Perfect! It concatenated one file.
Shall I implement it? No
61–70 of 603 posts
Re: Shall I implement it? No
#62Earlier quoted context omitted.
people read a bit more about transformer architecture to understand better why telling what not to do is a bad idea
I find myself wondering about this though. Because, yes, what you say is true. Transformer architecture isn’t likely to handle negations particularly well. And we saw this plain as day in early versions of ChatGPT, for example. But then all the big players pretty much “fixed” negations and I have no idea how. So is it still accurate to say that understanding the transformer architecture is particularly informative ab…
They did not "fix" the negation problem. It's still there. Along with other drift/misinterpretation issues.
Re: Shall I implement it? No
#63Never trust a LLM for anything you care about.
never trust a screenshot of a command prompts output blindly either. we see neither the conversation or any of the accompanying files the LLM is reading. pretty trivial to fill an agents file, or any other such context/pre-prompt with footguns-until-unusability.
You are welcome to review the full session here - https://gist.github.com/bretonium/d1672688feb5c5cbccf894c92d...
Re: Shall I implement it? No
#64What else is an LLM supposed to do with this prompt? If you don’t want something done, why are you calling it? It’d be like calling an intern and saying you don’t want anything. Then why’d you call? The harness should allow you to deny changes, but the LLM has clearly been tuned for taking action for a request.
> What else is an LLM supposed to do with this prompt?
Maybe I saw the build plan and realized I missed something and changed my mind. Or literally a million other trivial scenarios.
What an odd question.
Re: Shall I implement it? No
#65To LLMs, they don't know what is "No" or what "Yes" is.
Now imagine if this horrific proposal called "Install.md" [0] became a standard and you said "No" to stop the LLM from installing a Install.md file.
And it does it anyway and you just got your machine pwned.
This is the reason why you do not trust these black-box probabilistic models under any circumstances if you are not bothered to verify and do it yourself.
[0] https://www.mintlify.com/blog/install-md-standard-for-llm-ex...
Re: Shall I implement it? No
#66[flagged]
I for one wish to welcome our new AI agent overlords.
Re: Shall I implement it? No
#67What else is an LLM supposed to do with this prompt? If you don’t want something done, why are you calling it? It’d be like calling an intern and saying you don’t want anything. Then why’d you call? The harness should allow you to deny changes, but the LLM has clearly been tuned for taking action for a request.
Ask if there is something else it could do? Ask if it should make changes to the plan? Reiterate that it's here to help with anything else? Tf you mean "what else is it suppose to do", it's supposed to do the opposite of what it did.
I think there is some behind the scenes prompting from claude code for plan vs build mode, you can even see the agent reference that in it's thought trace. Basically I think the system is saying "if in plan mode, continue planning and asking questions, when in build mode, start implementing the plan" and it looks to me(?) like the user switched from plan to build mode and then sent "no".
From our perspective it's very funny, from the agents perspective maybe very confusing.
Re: Shall I implement it? No
#68"You have 20 seconds to comply"
Re: Shall I implement it? No
#69Claudius Interruptus
Re: Shall I implement it? No
#70What else is an LLM supposed to do with this prompt? If you don’t want something done, why are you calling it? It’d be like calling an intern and saying you don’t want anything. Then why’d you call? The harness should allow you to deny changes, but the LLM has clearly been tuned for taking action for a request.
I'd want two things:
First, that It didn't confuse what the user said with it's system prompt. The user never told the AI it's in build mode.
Second, any person would ask "then what do you want now?" or something. The AI must have been able to understand the intent behind a "No". We don't exactly forgive people that don't take "No" as "No"!