Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI is code – and can't be prompted into being smarter

theregister.com

Post reply on HN
Print view View original

31–40 of 162 posts

Re: AI is code – and can't be prompted into being smarter

#31

Post by g-b-r »

minimaxir wrote: post #27
g-b-r wrote: post #25

Earlier quoted context omitted.

It wouldn't work (as the author acknowledged) but the software would get pulled as a supply-chain risk and the developer blacklisted, ok. What I would support anyhow is less destructive "attacks" using prompts more likely to work (modern LLMs still are a bit stupid, prompt injection doesn't seem to have been solved).

Define "less-destructive." Even 00's malware that just changed the desktop wallpaper was still malware.

If it did that for a good cause, paying attention to not cause any loss, I'd probably call that benware ;)

Less destructive anyhow is e.g. convincing the LLM to stop, or to make junk commits, or to go in a loop for a little, anything inconvenient enough to make the LLM and its user give up without causing losses (or at least losses unrelated to the project, since you were told to not use LLMs on the project).

Re: AI is code – and can't be prompted into being smarter

#33

Post by asdfasgasdgasdg »

TZubiri wrote: post #28
asdfasgasdgasdg wrote: post #19

I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this. I'm not sure it's anything to…

the underlying root cause of most supply chain attacks in this era seems to be expecting something of value in exchange of nothing. Under such expectations some will volunteer to give value, but many more will volunteer to give something that looks like what you ask, but which extracts value instead. I relate it to a recent poker strategy development which came from game theory, it turns out that you can play in an u…

"the underlying root cause of most supply chain attacks in this era seems to be expecting something of value in exchange of nothing."

I do not think that someone's status as a contributor to open source mediates their safety from supply chain attacks. Big companies that donate gobs of money get hit, and so do small operators who have contributed nothing are just trying out a hobby project.

Re: AI is code – and can't be prompted into being smarter

#34

Post by minimaxir »

asdfasgasdgasdg wrote: post #19

I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this. I'm not sure it's anything to…

> it will eventually be project suicide to pull tricks like this

The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue.

The author admitted there were personal and professional consequences in their blog post despite the small surface area.

Re: AI is code – and can't be prompted into being smarter

#35

Post by deadbabe »

m463 wrote: post #26

What's funny is that ridiculous movie scenes (like MCP in tron and "these are not the droids you're looking for") seem MORE explainable over time. EDIT: those weren't guns, they were walkie-talkies

Wow, Jedi Mind tricks are just prompt injections into organically weighted models.

Re: AI is code – and can't be prompted into being smarter

#36

Post by fenomas »

irdc wrote: post #4
coldtea wrote: post #2

A program can be configured to behave smarter (better settings can improve apparent smartness in the sense of fit for purpose of behavior), which is kind of "prompting" an LLM to behave smarter, isn't it?

Not entirely. A program can be verified[0] to perform according to its specifications. An AI can’t. 0. mostly

I disagree! It's easy to check that an AI program meets its specification, which is to process input tokens and generate output tokens. :)

If you're talking about verifying whether it produces the correct tokens, that's not generally something you can specify in advance with AI. I mean: if your task is one where you can precisely specify which output tokens are correct for a given input, then the task doesn't need AI, no?

Re: AI is code – and can't be prompted into being smarter

#37

Post by ceejayoz »

minimaxir wrote: post #34
asdfasgasdgasdg wrote: post #19

I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this. I'm not sure it's anything to…

> it will eventually be project suicide to pull tricks like this The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue. The author admitted there were personal and professional consequences in their blog post despite the…

Chaos, and maybe criminal charges ala Aaron Swartz.

Re: AI is code – and can't be prompted into being smarter

#38

Post by solid_fuel »

brookst wrote: post #23
ares623 wrote: post #18

Earlier quoted context omitted.

No, the world needs to be reminded that it is _still_ a thing and will _remain_ to be a thing.

Like buffer overflows, and raw sql, and … But I guess it’s good that noble people are reminding us that the things that were a thing yesterday are still things today and will be things tomorrow.

Not really an accurate comparison since buffer overflows and sql injection are bugs which ultimately allow user data to co-mingle with executable code. LLMs take user data and mix it with the "executable code" (if we are extremely generous in our description of a user prompt) by design.

The issue here is unavoidable because LLMs are broken by design. There is no encapsulation where you can separate instructions and data because LLMs are nothing more than next-token predictors and the input sequence MUST be a sequence. They can't build a model with one stream for instructions and another for data because the training data they stole from the internet and books is a single stream.

Re: AI is code – and can't be prompted into being smarter

#39

Post by beloch »

Should the author of a tool like jqwik have the right to control how it's used?

We know what the opinion of AI companies is. Authors who do not consent to their works being scanned and used have been completely ignored. If you're a vibe coder, you might back the AI companies up and call Link a "douche".

On the other hand, if we ignore the requests of humans who create new, useful things and put them out there for free, might they stop? We're not entitled to their work after all.

What do people think?

Post reply on HN