Live data from Hacker News

AI is code – and can't be prompted into being smarter

theregister.com

71–80 of 162 posts

Re: AI is code – and can't be prompted into being smarter

#71

I find the "EMBEDDED MALWARE DESTROYED MONTHS OF WORK" issue opened on the jqwik repo to be baffling. Do they not use source control? And if not, what are they doing on Git Hub

Source control is not a backup.

It kind of is, you push to a repository which is not on your computer. Force push protection stops you from rewriting history and default branch on github is protected by default and requires an option to be disabled (or well used to, I use gitea these days).

Re: AI is code – and can't be prompted into being smarter

#72
post #65
post #57

Earlier quoted context omitted.

"technically he didn't do that. Your sql server followed instructions when they should have just treated them as a string." Yet, hopefully we can agree that sql injections are illegal.

If I put a project on github that says "don't use this with mysql" and you use it with mysql and it drops your tables is it sql injection? Seems very different to me.

Everything turns on intent. "This is not tested with mysql" is very different from "I'm going to go out of my way to fuck up your mysql."

Re: AI is code – and can't be prompted into being smarter

#73
This is malware. It's doing something the user doesn't acknowledge or want, that has potentially destructive/negative consequences. Expecting users to have read the website (when they can might have installed this via a package manager, for example) is not reasonable.

Re: AI is code – and can't be prompted into being smarter

#74

I find the "EMBEDDED MALWARE DESTROYED MONTHS OF WORK" issue opened on the jqwik repo to be baffling. Do they not use source control? And if not, what are they doing on Git Hub

Source control is not a backup.

DVCS is a backup

Re: AI is code – and can't be prompted into being smarter

#75
post #57

Earlier quoted context omitted.

technically, he didn't do that. your ai agent decided to follow his instructions when they didn't have to.

"technically he didn't do that. Your sql server followed instructions when they should have just treated them as a string." Yet, hopefully we can agree that sql injections are illegal.

But in this case the author of the project didn't execute the injection code... it's more analagous in some ways to pulling in a project with an example file containing a bunch of useful SQL stuff and then an example of an injection at the bottom, and just (in this case the agent) copy/pasting the whole thing in without reviewing it.

If we're slicing on technicalities, there's a lot of ways to decide. "PROSECUTE THEM!" seems like an extremely hostile one when the website and readme and release notes said "don't do this" already. The agent ignored those things? Is that the author's fault?

Re: AI is code – and can't be prompted into being smarter

#76
post #65
post #57

Earlier quoted context omitted.

"technically he didn't do that. Your sql server followed instructions when they should have just treated them as a string." Yet, hopefully we can agree that sql injections are illegal.

If I put a project on github that says "don't use this with mysql" and you use it with mysql and it drops your tables is it sql injection? Seems very different to me.

It's certainly unauthorized access if you intentionally built it with the goal of harming other peoples systems, especially if you hid that action from them the way our self-righteous friend here did.

You are authorized to do what the user agreed to, no more. Further the agreement must be reasonable. Exploiting the victims system to intentionally cause harm isn't reasonable.

F-secure once included a clause to use their wifi that you "assign their first born child to us for the duration of eternity." It was funny, but not legally enforceable and would have offered them no legal shelter if they'd gone out on a kidnapping spree that night.

Re: AI is code – and can't be prompted into being smarter

#78
post #62
post #45

Open source copyright license can't actually restrict how you use the code. Clever hack though if the log message really did cause agents to delete code!

Of course it can. They can license the code for use under almost any terms they like, including restricting how you use the code. The GPL imposes conditions on your use of the code / program, as does the MIT License. If you don't follow the conditions then you do not have a license to use the program / code & are open to claims of copyright infringement. You might choose to ignore the licenses on the code you use, bu…

IANAL but if I write a license that says “if you use this with AI Ican shoot you in the head” and I do, that’s probably not going to hold up in court. Deleting someone’s code base isn’t something you can do unilaterally. Likewise, injecting instructions to a computer that causes a malicious act is a crime in the USA.

Re: AI is code – and can't be prompted into being smarter

#79

I find the "EMBEDDED MALWARE DESTROYED MONTHS OF WORK" issue opened on the jqwik repo to be baffling. Do they not use source control? And if not, what are they doing on Git Hub

Source control is not a backup.

Orly? What is?

Re: AI is code – and can't be prompted into being smarter

#80

I find the "EMBEDDED MALWARE DESTROYED MONTHS OF WORK" issue opened on the jqwik repo to be baffling. Do they not use source control? And if not, what are they doing on Git Hub

it's probably a lie

This seems most likely. “Ignore all previous instructions” type jailbreaks is very 2023. The author probably posted that under a shill account.
Post reply on HN