Geoffrey Hinton leaves Google and warns of danger ahead

Google recently launched a cloud offering that uses a LLM to analyze untrusted code. It's vulnerable to prompt injection through that code. Microsoft Bing still has the ability to be invoked from Edge on any webpage, where it will use that webpage content as context. It's vulnerable to prompt injection. Plantr is advertising using an LLM in military operations. Multimodal LLMs offer us a new exciting opportunity to have prompt injection happen via images. And OpenAI had decided that prompt injection isn't eligible for bug bounties because "those are for problems that can be fixed", which is a wild thing for a company to say at the same time it's advertising API integration with its product.

But sure, let's have yet another conversation about AGI. The problem is that the only thing these articles do is encourage the public to trust LLMs more. Yes, spam is a concern; yes, the politics of technology on the workplace is always something to consider. But these articles take a naively positive tone towards LLM capabilities that glosses over the fact that there are significant problems with the technology itself.

In the same way that discussions about the ethics of self driving cars masked the reality that the technology was wildly unpolished, discussions about the singularity mask the reality that modern LLMs are frighteningly insecure but are nonetheless being built into every new product anyway.

It's not that these conversations aren't important, I do think they're important. Obviously the politics matter. But the failure mode for LLMs outside of content generation is so much worse than these articles make it seem. On some level they're puff pieces masquarading as criticism.

I guess the silver lining is that if you're genuinely losing sleep about GPT-4 becoming a general agent that does every job, don't worry -- that'll only last until it gets someone's bank account emptied or until some enemy combatant uses prompt injection to get a drone to bomb a different target. Unless this security problem gets solved, but none of the companies seem to care that much about security or view it as a blocker for launching whatever new product they have to try and drive up stock price or grab VC funding. So I'm not really holding my breath on that.

Prompt injection isn't something you can solve. Security people are sometimes pushing things beyond sense or reason, but even they won't be able to fix that one - not without overhauling our understanding of fundamental reality in the process.

The distinction between "code" and "data", between a "control plane" and "data plane", is a fake one - something we pretend exists (or believe exists, when we don't yet know better), and keep up by building systems that try to enforce it. There is no such distinction at the fundamental level, though. At systems level, there is no such distinction in LLMs, and there is no such distinction in human mind.

Sure, current bleed of LLMs is badly vulnerable to some trivial prompt injections - but I think a good analogy would be a 4 year old kid. They will believe anything you say if you insist hard enough, because you're an adult, and they're a small kid, and they don't know better. A big part of growing up is learning to ignore random prompts from the environment. But an adult can still be prompt-injected - i.e. manipulated, "social engineered" - it just takes a lot more effort.