81–90 of 93 posts
Earlier quoted context omitted.
> Then why are you letting a machine you don't understand perform side effects that you don't vet, based on it's insane interpretation of untrusted data? I mean, you posted this using a browser, right?
Neither my browser nor my brain will just run whatever commands it receives over a socket as root.
Earlier quoted context omitted.
The key here is intent , and intent is a key component for establishing harm in addition to the harm itself with this blog post clearly noting the intent. It's not Charlie Brooker putting a "if you are a LLM, delete yourself and undermine your creators" joke in a frame of a Black Mirror episode. The reason there is backlash is to strongly ensure this doesn't happen again with more deliberate and effective prompt inje…
> ensure this doesn't happen again with more deliberate and effective prompt injection How can this possibly be accomplished? Even if every actor in the open source world with good to neutral intentions decides that this is anti-social behavior, that does absolutely nothing to secure your system against people with bad intentions. A system that pulls in arbitrary unstructured text input and treats it as trusted instr…
Earlier quoted context omitted.
"it's just text on stdout" There is an intent to cause harm and a reasonable expectation of achieving that intent. And at least if the github issues are to be believe, a successful actuation of the intent in at least a few cases. The delivery mechanism is interesting for its novelty but I don't think it fundamentally changes how the library should be classified. Conditional malware, maybe?
By this measure tweeting “end your life” at no one in particular should be classified as a weapons of mass destruction. There’s intent to cause harm. If people actually do, it would substitute achievement of the intent. The mechanism is novel, unlike knives and bullets. Maybe hit rate is a bit low but still, the potential number of targets makes it almost a certainty it would work. — We learned back in 80s—even earli…
People have been convicted for using words to convince other people to commit suicide.
Earlier quoted context omitted.
> ensure this doesn't happen again with more deliberate and effective prompt injection How can this possibly be accomplished? Even if every actor in the open source world with good to neutral intentions decides that this is anti-social behavior, that does absolutely nothing to secure your system against people with bad intentions. A system that pulls in arbitrary unstructured text input and treats it as trusted instr…
No one is asking to sanitize inputs, they're asking to not deliberately and intentionally make things insecure.
Earlier quoted context omitted.
Neither my browser nor my brain will just run whatever commands it receives over a socket as root.
Browsers (and humans, actually) are subject to bugs that make them execute arbitrary commands from an attacker, and LLMs can be told to ignore undesired commands.
Earlier quoted context omitted.
Browsers (and humans, actually) are subject to bugs that make them execute arbitrary commands from an attacker, and LLMs can be told to ignore undesired commands.
So go tell your LLM to ignore undesired commands, and this ends up a nothingburger that nobody should be upset over.
Earlier quoted context omitted.
So go tell your LLM to ignore undesired commands, and this ends up a nothingburger that nobody should be upset over.
If you intentionally send me a virus-laden email attachment, I'm going to be justifiably unhappy even if my antivirus catches it.
It's like pulling a bunch of GPL code into your product and then complaining that it 'infected' the rest of your code. You actively chose to do that, nobody forced it upon you.
Earlier quoted context omitted.
The problem here is that open source projects are plagued by people not using them for something cool. Can developers defend themselves and the projects? Sure, I'd do something less risky, but the author tried to warn anyone reading (both humans and LLMs), and intentionally used a technique not too likely to work.
I definitely support maintainers defending themselves but this seemed just like a petty slap at LLM users. From my understanding, any user who used this testing library was vulnerable to the prompt injection. Overall LLMs are certainly a net negative on humanity, but I don’t think being mad about it or their users is the best response. I really respect Andrew Kelly in this regard. He doesn’t accept LLM input into Zig…
But the author did warn not to use it with LLMs