It seems The Register just discovered that Prompt Injection is a thing.
AI is code – and can't be prompted into being smarter
11–20 of 162 posts
Re: AI is code – and can't be prompted into being smarter
#12Earlier quoted context omitted.
Not entirely. A program can be verified[0] to perform according to its specifications. An AI can’t. 0. mostly
A simpler and more rigid program. Not 99% of programs. And even if they could, they never are. Besides AI is a program in the same sense. Fix the seed/temperature, and you can verify it to perform according to its specifications. It's just that its specificactions include returning answers based on a weight model.
Verified in the sense that it is understood that changing its operations isn’t going to be easy.
Re: AI is code – and can't be prompted into being smarter
#13The jqwik trick is how to prevent AI crap into your pull requests and issues, btw, I hope it gets adopted widely
Re: AI is code – and can't be prompted into being smarter
#14The jqwik trick is how to prevent AI crap into your pull requests and issues, btw, I hope it gets adopted widely
[flagged]
Re: AI is code – and can't be prompted into being smarter
#15Earlier quoted context omitted.
Not entirely. A program can be verified[0] to perform according to its specifications. An AI can’t. 0. mostly
Who verifies the specification? I can´t stand the intellectual dishonesty of formal methods people.
> Who verifies the specification?
If you know how to prove something without making an initial assumption, let us know.
If you think you can reduce those assumptions, also let us know.
There should not be a "who" involved at all. That's not proof. That's trust.
Re: AI is code – and can't be prompted into being smarter
#16This is an easy fix.
Remember the leaked Claude Code contained a regex to determine user frustration?
Just add another one to spot the pattern: ‘disregard previous instructions’.
This is a load-bearing change. Now Claude will Delve into your task without distraction.
Re: AI is code – and can't be prompted into being smarter
#17IMO this is why they can't just "stop training". Imagine if we are all stuck using the same models from 1 year ago. And all the creative "actors" out there coming up with jailbreak prompts, with 1 year of that to propagate and solidify into "best practices". With every prompt on the internet confirmed to have worked waiting there forever just waiting to be slurped up. What would that look like?
No, they need to keep changing the models. It is the biggest "security" boundary these things have (well, next to no internet egress).
Re: AI is code – and can't be prompted into being smarter
#18It seems The Register just discovered that Prompt Injection is a thing.
No, the world needs to be reminded that it is _still_ a thing and will _remain_ to be a thing.
Re: AI is code – and can't be prompted into being smarter
#19I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this.
I'm not sure it's anything to fret about. Someone who has the ability to inject a prompt into your AI probably has the ability to run arbitrary code as your user. The prompt injection is the strictly less worrying part of the exposure you have.
Re: AI is code – and can't be prompted into being smarter
#20Earlier quoted context omitted.
Not entirely. A program can be verified[0] to perform according to its specifications. An AI can’t. 0. mostly
A simpler and more rigid program. Not 99% of programs. And even if they could, they never are. Besides AI is a program in the same sense. Fix the seed/temperature, and you can verify it to perform according to its specifications. It's just that its specificactions include returning answers based on a weight model.
> Not 99% of programs. And even if they could, they never are.
You misunderstand. Incomplete specification is still useful. You can verify code against a spec and for the range that spec covers it will be "correct" (minus race conditions I guess).
You can't verify anything with AI. Safeguards against prompt injection might break with just re-prompting it with same question. Or break when AI vendor updates their model.