181–190 of 378 posts
Aside: I've found that 'not'[0] isn't something that LLMs can really understand. Like, with us humans, we know that if you use a 'not', then all that comes after the negation is modified in that way. This is a really strong signal to humans as we can use logic to construct meaning. But with all the matrix math that LLMs use, the 'not' gets kinda lost in all the other information. I think this is because with a modern…
I’ve hit this! In my otherwise wildly successful attempt to translate a Haskell codebase to Clojure [0], Claude at one point asks: [Claude:] Shall I commit this progress? [some details about what has been accomplished follow] Then several background commands finish (by timeout or completing); Claude Code sees this as my input, thinks I haven’t replied to its question, so it answers itself in my name: [Claude:] Yes, g…
Aside: I've found that 'not'[0] isn't something that LLMs can really understand. Like, with us humans, we know that if you use a 'not', then all that comes after the negation is modified in that way. This is a really strong signal to humans as we can use logic to construct meaning. But with all the matrix math that LLMs use, the 'not' gets kinda lost in all the other information. I think this is because with a modern…
Do you have evals for this claim? I don't really experience this
It's enough of a problem that it's in my private benchmarks for all new models.
Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…
I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.
Models/Agents need a narrow set of things they are allowed to actually trigger, with real security policies, just like people.
You can mitigate agent->agent triggers by not allowing direct prompting, but by feeding structured output of tool A into agent B.
Earlier quoted context omitted.
I don't think so, feels like the wrong side is getting attention. Degrading the experience for humans (in one tool) because the bots are prone to injection (from any tool) . Terraform is used outside of agents; somebody surely finds the reminder helpful. If terraform were to abide, I'd hope at the very least it would check if in a pipeline or under an agent. This should be obvious from file descriptors/env. What abou…
"Run terraform apply plan.out next" in this context is a prompt injection for an LLM to exactly the same degree it is for a human. Even a first party suggestion can be wrong in context, and if a malicious actor managed to substitute that message with a suggestion of their own, humans would fall for the trick even more than LLMs do. See also: phishing.
Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…
I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.
Because LLMs are inherently designed to interface with humans through natural language. Trying to graft a machine interface on top of that is simply the wrong approach, because it is needlessly computationally inefficient, as machine-to-machine communication does not - and should not - happen through natural language.
The better question is how to design a machine interface for communicating with these models. Or maybe how to design a new class of model that is equally powerful but that is designed as machine first. That could also potentially solve a lot of the current bottlenecks with the availability of computer resources.
Earlier quoted context omitted.
The real issue is expecting an LLM to be deterministic when it's not.
they are deterministic, open a dev console and run the same prompt two times w/ temperature = 0