Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI is code – and can't be prompted into being smarter

theregister.com

Post reply on HN
Print view View original

141–150 of 162 posts

Re: AI is code – and can't be prompted into being smarter

#141

Post by vips7L »

minimaxir wrote: post #21
g-b-r wrote: post #13

The jqwik trick is how to prevent AI crap into your pull requests and issues, btw, I hope it gets adopted widely

The jqwik trick wouldn't work in practice because modern LLMs aren't that stupid, which makes the whole thing pointlessly performative. If someone else tried to do the same thing again with a more popular/widely-used software, a) the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing.

> the software would just get pulled as a supply-chain risk and b) the developer would likely be blacklisted. Again, accomplishing nothing.

Oh no the people I don’t want using my software aren’t going to use it. The horror.

Re: AI is code – and can't be prompted into being smarter

#142

Post by infinite_spin »

ceejayoz wrote: post #134
infinite_spin wrote: post #99

Earlier quoted context omitted.

Aaron Swartz didn't code malware into a project with the stated goal of destroying data.. and this has nothing to do with copyright infringement or an overzealous federal agency. How does this situation relate to what he went through?

It relates because they went after him for much less.

I sincerely ask that you do not trivialize what happened to Aaron Swartz by linking what happened to him to any criminal actions on his part. He did nothing wrong.

Re: AI is code – and can't be prompted into being smarter

#143

Post by ceejayoz »

infinite_spin wrote: post #142
ceejayoz wrote: post #134

Earlier quoted context omitted.

It relates because they went after him for much less.

I sincerely ask that you do not trivialize what happened to Aaron Swartz by linking what happened to him to any criminal actions on his part. He did nothing wrong.

That’s precisely the point.

If his non-wrong acts could be criminally prosecuted like that, this case - intentional damage! - is even riskier.

Re: AI is code – and can't be prompted into being smarter

#144

Post by tcp_handshaker »

sublinear wrote: post #15
tcp_handshaker wrote: post #5

Earlier quoted context omitted.

Who verifies the specification? I can´t stand the intellectual dishonesty of formal methods people.

> Who verifies the specification? If you know how to prove something without making an initial assumption, let us know. If you think you can reduce those assumptions, also let us know. There should not be a "who" involved at all. That's not proof. That's trust.

[flagged]

Re: AI is code – and can't be prompted into being smarter

#145

Post by brookst »

solid_fuel wrote: post #38
brookst wrote: post #23

Earlier quoted context omitted.

Like buffer overflows, and raw sql, and … But I guess it’s good that noble people are reminding us that the things that were a thing yesterday are still things today and will be things tomorrow.

Not really an accurate comparison since buffer overflows and sql injection are bugs which ultimately allow user data to co-mingle with executable code. LLMs take user data and mix it with the "executable code" (if we are extremely generous in our description of a user prompt) by design. The issue here is unavoidable because LLMs are broken by design. There is no encapsulation where you can separate instructions and d…

While I agree that LLMs have yet again surfaced the “new tech fails to separate data and control” issue that affected everything from pay phones to SQL, I disagree that there’s something different that prevents the introduction of separate planes.

That “stolen” training data, most of which itself was stolen from older works, does not include user prompts. It is data, not control.

We will see models with annotations for whether a token is part of user prompt, and other ways as well.

You’re obviously passionate about the subject but as someone who works in the field, I assure you there is no now-and-forever requirement for a single stream with no metadata about tokens. We will positively see control and data separated just like they were for phones and databases.

Re: AI is code – and can't be prompted into being smarter

#146

Post by DangitBobby »

fennecbutt wrote: post #125

Can't even be bothered greasing the article because the attention grabbing headline simply isn't true. (edit: oh it's the register, so absolute trash) It was already YEARS ago that they found that certain things such as the time of year (December vs. start of January) had an impact on reasoning effort. Until we're training models such that the undesirable human patterns aren't picked up from training data there will…

You aren't missing anything except an embarrassing amount of ego on display in the article.

> the techbro botlickers tend to ignore that sort of thing

(admitting up front that users won't see the notice not to upgrade from 1.9 to 1.10)

> Naturally, this sort of "developer" – we use the word fairly loosely here, you understand – doesn't read the code first. That would ruin the vibe, man.

> You can probably guess what happened next: suddenly, there were a lot of very unhappy ChatNPCs

> In his follow-up blog post this week, The Jqwik Anti-AI Affair, Link innocently (or perhaps ever so slightly disingenuously) explains: "The line was not visible when you looked at it in an emulated terminal. I added this fade-out feature because I personally do not want to see it."

That's not at all nefarious huh

> Oh dear. How sad. Never mind.

> Prompt fondlers

Re: AI is code – and can't be prompted into being smarter

#147

Post by krupan »

Quarrel wrote: post #62
krupan wrote: post #45

Open source copyright license can't actually restrict how you use the code. Clever hack though if the log message really did cause agents to delete code!

Of course it can. They can license the code for use under almost any terms they like, including restricting how you use the code. The GPL imposes conditions on your use of the code / program, as does the MIT License. If you don't follow the conditions then you do not have a license to use the program / code & are open to claims of copyright infringement. You might choose to ignore the licenses on the code you use, bu…

Using software is not the same as copying the code. Copyright law only covers making and distributing copies of things. It's a subtlety of the law.

By default (in the US) you are not permitted to copy someone else's work (with some exceptions for "fair use") without the copyright holders permission.

Software copyright holders generally give you a copy of their software only if you agree with their terms (it's a contract agreement, or license. Their terms usually bind you to not use the software in certain ways and to not make any copies of the copy you have been given. If you break that contract then you have no permission to have a copy of the software and you are in violation of copyright law.

None of the Open Source licenses restrict how you use the software. If you create your own license that does restrict use then by definition it is not an Open Source license. Open Source licenses do put restrictions on copies you make and distribute (some licenses impose more restrictions than others).

Re: AI is code – and can't be prompted into being smarter

#148

Post by mapontosevenths »

ethin wrote: post #89
mapontosevenths wrote: post #63

Earlier quoted context omitted.

I'm fairly certain he is wrong. A lot of folks lean on Shenk, and I think he does in that video though I haven't watched it all. Shenk was overturned by Breandenburg v. Ohio, and in in it they are explicit that shouting fire in a crowded theater is very much one of the only kinds of speech that IS restricted. They literally use that example in the decision. Quote: "The example usually given by those who would punish…

Given that even Wikipedia effectively restates what he says, I'm pretty sure he's correct here: > Ultimately, whether it is legal in the United States to falsely shout "fire" in a theater depends on the circumstances in which it is done and the consequences of doing it. The act of shouting "fire" when there are no reasonable grounds for believing one exists is not in itself a crime, and nor would it be rendered a cri…

You're right that it's an almost impossibly high bar to clear, but saying that you can't be prosecuted for it or that it's not illegal is pretty darn misleading, or at least glossing over some pretty important nuance. It's literally the example the Supreme court chose to use as an example of one of the few times you CAN be prosecuted. Telling everyone the opposite isn't helpful.

As I said, I haven't watched Legal Eagles full video, but I don't like that ever since that video came out everyone on the internet tries to correct anyone who uses the phrase even in it's colloquial sense. Maybe the source video covers the nuance (I wouldn't know), but the folks on the internet "umm acshually"ing everyone seldom do.

Re: AI is code – and can't be prompted into being smarter

#149

Post by mapontosevenths »

easyThrowaway wrote: post #124
asdfasgasdgasdg wrote: post #19

I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this. I'm not sure it's anything to…

The most controversial part is probably that something like jqwik could be a dependency you're not even aware of. You could be asking your agent to fully analyze a project to isolate a bug and in the right situation that would probably trigger the exploit.

Thanks for this. A lot of folks here going "well you read the license agreement, didn't you!?!" As if that was even an option when you update some package via NPM or whatever.

Re: AI is code – and can't be prompted into being smarter

#150

Post by ceejayoz »

mapontosevenths wrote: post #148
ethin wrote: post #89

Earlier quoted context omitted.

Given that even Wikipedia effectively restates what he says, I'm pretty sure he's correct here: > Ultimately, whether it is legal in the United States to falsely shout "fire" in a theater depends on the circumstances in which it is done and the consequences of doing it. The act of shouting "fire" when there are no reasonable grounds for believing one exists is not in itself a crime, and nor would it be rendered a cri…

You're right that it's an almost impossibly high bar to clear, but saying that you can't be prosecuted for it or that it's not illegal is pretty darn misleading, or at least glossing over some pretty important nuance. It's literally the example the Supreme court chose to use as an example of one of the few times you CAN be prosecuted. Telling everyone the opposite isn't helpful. As I said, I haven't watched Legal Eag…

The seminal work in this world is from Popehat (Ken White) explaining the complexities of it all. https://www.popehat.com/p/the-first-amendment-isnt-absolute
Post reply on HN