Earlier quoted context omitted.
> If you want the AI to do anything useful, you need to be able to trust it with the access to useful things. Sandboxing doesn't solve this. By default, AI cannot be trusted because it is not deterministic. You can't audit what the output of any given prompt is going to be to make sure its not going to rm -rf / We need some form of behavioral verification/auditing with guarantees that any input is proven to not produ…
Determinism is an absolute red herring. A correct output can be expressed in an infinite amount of ways, all of them valid. You can always make an LLM give deterministic outputs (with some overhead), that might bring you limited reproducibility, but that won't bring you correctness. You need correctness, not determinism. >We need some form of behavioral verification/auditing with guarantees that any input is proven t…
I suspect that it may merely be infeasible, not strictly impossible. There has been work on automatically proving that an ANN satisfies certain properties (iirc e.g. some kinds of robustness to some kinds of adversarial inputs, for handling images).
It might be possible (though infeasible) to have an effective LLM along with a proof that e.g. it won't do anything irreversible when interacting with the operating system (given some formal specification of how the operating system behaves).
But, yeah, in practice I think you are correct.
It makes more sense to put the LLM+harness in an environment which ensures you can undo whatever it does if it messes things up, than to try to make the LLM be such that it certainly won't produce outputs that would mess things up in a way that isn't easily revertible, even if it does turn out that the latter is in principle possible.