281–290 of 378 posts
I have seen this when approaching ~30% context window remaining. There was a big bug in the Voice MCP I was using that it would just talk to itself back and forth too.
I'll have it create a handoff document well before it hits 50% and it seems to help.
Most of our team has moved to cursor or codex since the March downgrade (https://github.com/anthropics/claude-code/issues/42796)
Claude Code is actually far from the best harness for Claude, ironically...
JetBrains' AI Assistant with Claude Agent is a much better harness for Claude.
Earlier quoted context omitted.
The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.
As the article says: this doesn’t necessarily appear to be a problem in the LLM, it’s a problem in Claude code. Claude code seems to leave it up to the LLM to determine what messages came from who, but it doesn’t have to do that. There is a deterministic architectural boundary between data and control in Claude code, even if there isn’t in Claude.
Earlier quoted context omitted.
The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.
It’s easier not to have that separation, just like it was easier not to separate them before LLMs. This is architectural stuff that just hasn’t been figured out yet.
Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…
[1]: https://en.wikipedia.org/wiki/Three_Laws_of_Robotics
[2]: https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
Earlier quoted context omitted.
That's just general context rot, and the models do all sorts of off the rails behavior when the context is getting too unwieldy. The whole breakthrough with LLM's, attention, is the ability to connect the "not" with the words it is negating.
This doesn't mean there's no subtle accuracy drop on negations. Negations are inherently hard for both humans and LLMs because they expand the space of possible answers, this is a pretty well studied phenomenon. All these little effects manifest themselves when the model is already overwhelmed by the context complexity, they won't clearly appear on trivial prompts well within model's capacity.
Like, in Latin, the verb is at the end. In that, it's structured like how Yoda speaks.
So, especially with Cato, you kinda get lost pretty easy along the way with a sentence. The 'not's will very much get forgotten as you're waiting for the verb.
Earlier quoted context omitted.
This was a problem with early telephone lines which was easy to exploit (see Woz & Jobs Blue Box). It got solved by separating the voice and control pane via SS7. Maybe LLMs need this separation as well
This is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.
Humans have a problem distinguishing "John from Microsoft" from somebody just claiming to be John from Microsoft. The reason why scamming humans is (relatively) hard is that each human is different. Discovering the perfect tactic to scam one human doesn't necessarily scale across all humans.
LLMs are the opposite; my Chat GPT is (almost) the same as your Chat GPT. It's the same model with the same system message, it's just the contexts that differ. This makes LLM jailbreaks a lot more scalable, and hence a lot more worthwhile to discover.
LLMs are also a lot more static. With people, we have the phenomenon of "banner blindness", which LLMs don't really experience.
Earlier quoted context omitted.
This was a problem with early telephone lines which was easy to exploit (see Woz & Jobs Blue Box). It got solved by separating the voice and control pane via SS7. Maybe LLMs need this separation as well
This is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.
Earlier quoted context omitted.
It’s easier not to have that separation, just like it was easier not to separate them before LLMs. This is architectural stuff that just hasn’t been figured out yet.
There is a system prompt, but most LLMs don't seem to "enforce" it enough.
Earlier quoted context omitted.
This is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.
I think this is fundamental to any technology, including human brains. Humans have a problem distinguishing "John from Microsoft" from somebody just claiming to be John from Microsoft. The reason why scamming humans is (relatively) hard is that each human is different. Discovering the perfect tactic to scam one human doesn't necessarily scale across all humans. LLMs are the opposite; my Chat GPT is (almost) the same…
The foundation of LLMs is Attention.