21–30 of 162 posts
The jqwik trick is how to prevent AI crap into your pull requests and issues, btw, I hope it gets adopted widely
If someone else tried to do the same thing again with a more popular/widely-used software, a) the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing.
It seems The Register just discovered that Prompt Injection is a thing.
No, the world needs to be reminded that it is _still_ a thing and will _remain_ to be a thing.
But I guess it’s good that noble people are reminding us that the things that were a thing yesterday are still things today and will be things tomorrow.
I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this. I'm not sure it's anything to…
The jqwik trick is how to prevent AI crap into your pull requests and issues, btw, I hope it gets adopted widely
The jqwik trick wouldn't work in practice because modern LLMs aren't that stupid, which makes the whole thing pointlessly performative. If someone else tried to do the same thing again with a more popular/widely-used software, a) the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing.
What I would support anyhow is less destructive "attacks" using prompts more likely to work (modern LLMs still are a bit stupid, prompt injection doesn't seem to have been solved).
EDIT: those weren't guns, they were walkie-talkies
Earlier quoted context omitted.
The jqwik trick wouldn't work in practice because modern LLMs aren't that stupid, which makes the whole thing pointlessly performative. If someone else tried to do the same thing again with a more popular/widely-used software, a) the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing.
It wouldn't work (as the author acknowledged) but the software would get pulled as a supply-chain risk and the developer blacklisted, ok. What I would support anyhow is less destructive "attacks" using prompts more likely to work (modern LLMs still are a bit stupid, prompt injection doesn't seem to have been solved).
I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this. I'm not sure it's anything to…
Under such expectations some will volunteer to give value, but many more will volunteer to give something that looks like what you ask, but which extracts value instead.
I relate it to a recent poker strategy development which came from game theory, it turns out that you can play in an unexploitable manner, but it will usually result in ties, and lost time and money to rake, and theoretically any attempt to exploit another player, leaves you exploitable to another player. The classical example is rock paper scissors, unexploitable strategy is to play randomly with p=1/3 for each choice, however if one really wishes to win more often than their opponent, they have to guess, and if in that guessing they choose an option with 100% certainty, they become exploitable to someone choosing another option with 100% certainty.
In effect the very act of attempting to extract value from free software, is the very act that leaves one vulnerable to being extracted value from.
You’re not making performance gains, as often as you’re getting back out of the way.
This is an easy fix. Remember the leaked Claude Code contained a regex to determine user frustration? Just add another one to spot the pattern: ‘disregard previous instructions’. This is a load-bearing change. Now Claude will Delve into your task without distraction.