>
You're overcomplicating a thing that is simple -- don't use in-band control signaling.On the contrary, I'm claiming that this "simplicity" is an illusion. Reality has only one band.
> It's been the same problem since whistling for long-distance, with the same solution of moving control signals out of the data stream.
"Control signals" and "data stream" are just... two data streams. They always eventually mix.
> The same solution, hard isolation, instantly solves the problem: you have to render control inexpressible in the in-band alphabet.
This isn't something that exist in nature. We don't build machines out of platonic shapes and abstract math - we build them out of matter. You want such rules like "separation of data and code", "separation of control-data and data-data", and "control-data being inexpressible in data-data alphabet" to hold? You need to design a system so constrained, as to behave this way - creating a faux reality within itself, where those constraints hold. But people keep forgetting - this is a faux reality. Those constraints only hold within it, not outside it[0], and to the extent you actually implemented what you thought you did (we routinely fuck that up).
I start to digress, so to get back to the point: such constraints are okay, but they by definition limit what the system could do. This is fine when that's what you want, but LLMs are explicitly designed to not be that. LLMs are built for one purpose - to process natural language like we do. That's literally the goal function used in training - take in arbitrary input, produce output that looks right to humans, in fully general sense of that[1].
We've evolved to function in the physical reality - not some designed faux-reality. We don't have separate control and data channels. We've developed natural language to describe that reality, to express ourselves and coordinate with others - and natural language too does not have any kind of control and data separation, because our brains fundamentally don't implement that. More than that, our natural language relies on there being no such separation. LLMs therefore cannot be made to have that separation either.
We can't have it both ways.
--
[0] - The "constraints only apply within the system" part is what keeps tripping people over. You may think your telegraph cannot possibly be controlled over the data wire - it really doesn't even parse the data stream, literally just forwards it as-is, to a destination selected on another band. What you don't know is, I looked up the specs of your telegraph, and figured out that if I momentarily plug a car battery to the signal line, it'll briefly overload a control relay in your telegraph, and if I time this right, I can make the telegraph switch destinations.
(Okay, you treat it as a bug and add some hardware to eliminate "overvoltage events" from what can be "expressed in the in-band alphabet". But you forgot that the control and data wires actually run close to each other for a few meters - so let me introduce you to the concept of electromagnetic induction.)
And so on, and so on. We call those things "side channels", and they're not limited to exploiting physics; they're just about exploiting the fact that your system is built in terms of other systems with different rules.
[1] - Understanding, reasoning, modelling the world, etc. all follow directly from that - natural language directly involves those capabilities, so having or emulating them is required.