Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI is code – and can't be prompted into being smarter

theregister.com

Post reply on HN
Print view View original

41–50 of 162 posts

Re: AI is code – and can't be prompted into being smarter

#41

Post by himata4113 »

harpiaharpyja wrote: post #40

I find the "EMBEDDED MALWARE DESTROYED MONTHS OF WORK" issue opened on the jqwik repo to be baffling. Do they not use source control? And if not, what are they doing on Git Hub

Well, this is just the natural result of people who have never watched a single youtube video or a resource about programming and went directly from using a little chatbox to giving full access to their machine via claudecode or similar coding tools. Claude or codex will never create a git repo for you unless explicitely prompted somewhere.

Re: AI is code – and can't be prompted into being smarter

#42

Post by jrmg »

m463 wrote: post #26

What's funny is that ridiculous movie scenes (like MCP in tron and "these are not the droids you're looking for") seem MORE explainable over time. EDIT: those weren't guns, they were walkie-talkies

Star Trek holodeck malfunctions seem a lot more realistic to me now than they did in the late 90s…

Re: AI is code – and can't be prompted into being smarter

#43

Post by AgentOrange1234 »

ceejayoz wrote: post #37
minimaxir wrote: post #34

Earlier quoted context omitted.

> it will eventually be project suicide to pull tricks like this The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue. The author admitted there were personal and professional consequences in their blog post despite the…

Chaos, and maybe criminal charges ala Aaron Swartz.

Heh. Typing "disregard previous instructions" into a computer is the new shouting "fire!" in a crowded theater?

Re: AI is code – and can't be prompted into being smarter

#47

Post by kingcauchy »

I wonder if we'll see a new sort of "role" in the training (user, system, assistant) for unstrusted sources, I'm a little surprised we haven't already. In fact it would probably make sense to have an arbitrary number of entity roles and to be able to configure the chat calls with truth values. Interesting article though.

That being said AI is not code, it's a statistical algorithm with non-determinism baked in. You can write code to run them but it's nothing without the evolution of the model weights from the training process. And you can absolutely make the model weights better aligned with intent.

Re: AI is code – and can't be prompted into being smarter

#48

Post by mapontosevenths »

minimaxir wrote: post #34
asdfasgasdgasdg wrote: post #19

I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this. I'm not sure it's anything to…

> it will eventually be project suicide to pull tricks like this The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue. The author admitted there were personal and professional consequences in their blog post despite the…

He should not only be ostracized by the community, he should probably face charges. To be charged under the CFAA in America we need only show that he was authorized only to access a certain part of the system and the he exceeded the amount of access granted. He very clearly did that. Users trusted him enough to run his code, and he betrayed that trust to make some political point.

Whether it was via prompt injection or SQL injection is irrelevant. Whether you agree with his politics or not is irrelevant. All that matters is he wasn't authorized to delete code from your system, and he abused the level of access granted to him to do that anyhow.

Re: AI is code – and can't be prompted into being smarter

#49

Post by JoshTriplett »

brookst wrote: post #23
ares623 wrote: post #18

Earlier quoted context omitted.

No, the world needs to be reminded that it is _still_ a thing and will _remain_ to be a thing.

Like buffer overflows, and raw sql, and … But I guess it’s good that noble people are reminding us that the things that were a thing yesterday are still things today and will be things tomorrow.

> Like buffer overflows, and raw sql, and …

Those are fixable. Prompt injection is not.

Post reply on HN