Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Jqwik Anti-AI Affair

blog.johanneslink.net

Post reply on HN
Print view View original

61–70 of 93 posts

Re: The Jqwik Anti-AI Affair

#61

Post by solid_fuel »

skeledrew wrote: post #60
solid_fuel wrote: post #59

Earlier quoted context omitted.

Ah, this argument again. https://news.ycombinator.com/item?id=48534984

Doesn't apply. But hey, let's see another maintainer try this, and on a very public project, and someone using a LLM on that project get hit in such a bad way that they decide to go legal. Be interesting to see how that would proceed.

> But hey, let's see another maintainer try this, and on a very public project, and someone using a LLM on that project get hit in such a bad way that they decide to go legal.

I agree that would be very funny.

Re: The Jqwik Anti-AI Affair

#62

Post by gmerc »

asdfasgasdgasdg wrote: post #22
gmerc wrote: post #16

Earlier quoted context omitted.

If your supply chain is predicated on executing all text it reads as instructions, you deserve every single thing coming for you.

I actually do not think that this is fundamentally much more risky than the basic type of supply chain attack that already exists in code form. You actually have a lot less exposure, because when you give people the ability to run code on your computer, it works deterministically, whereas most AIs are becoming hardened to the sort of prompt injection attack we are discussing here. To put it another way, AI prompt inj…

Every app including a transformer is suddenly vulnerable to RCE from text.

Re: The Jqwik Anti-AI Affair

#63

Post by aselimov3 »

KronisLV wrote: post #41
QuadmasterXLII wrote: post #21

Earlier quoted context omitted.

what is anti ai psychosis? never heard of this.

From what I've seen, AI psychosis is blindly trusting the output of LLMs and sometimes trusting them instead of one's own critical thinking skills. Sometimes this leads to delusions, paranoia and spiraling, especially when combined with anthropomorphizing the technology and not knowing its limitations. Things such as ascribing sentience or consciousness to a machine that largely just predicts tokens. It gets especial…

Yea good explanation of what I meant. I understand the moral/ethical arguments against LLMs and they are compelling but the actions of the maintainer above seemed just like a petty swing at LLMs just for the sake of it. Not very rational in my opinion.

I’m trying to minimize my use of LLMs because I think they are harmful personally but I don’t get mad at people that use them (unless they are just spamming slop but then I just ignore it)

Re: The Jqwik Anti-AI Affair

#64

Post by yjftsjthsd-h »

vkou wrote: post #52
asdfasgasdgasdg wrote: post #49

Earlier quoted context omitted.

I think it is extremely rare to vet every single line of one's dependencies. Especially lines that are intentionally hidden from the terminal using escape sequences. Do you review the diffs of all projects you depend on to check for the injection of malware? If so, my hat is off to you and also how do you get anything else done?

Then why are you letting a machine you don't understand perform side effects that you don't vet, based on it's insane interpretation of untrusted data?

> Then why are you letting a machine you don't understand perform side effects that you don't vet, based on it's insane interpretation of untrusted data?

I mean, you posted this using a browser, right?

Re: The Jqwik Anti-AI Affair

#65

Post by aselimov3 »

g-b-r wrote: post #38
aselimov3 wrote: post #12

Earlier quoted context omitted.

I don’t think so but maybe? I do use them in daily work so I might be compromised. But I also generally dislike their impact on humanity and try to limit my use where feasible for my own brain’s sake. Personally I think Andrew Kelly’s take is the best. Basically not interested in LLMs but if someone uses them to do something cool then cool I guess?

The problem here is that open source projects are plagued by people not using them for something cool. Can developers defend themselves and the projects? Sure, I'd do something less risky, but the author tried to warn anyone reading (both humans and LLMs), and intentionally used a technique not too likely to work.

I definitely support maintainers defending themselves but this seemed just like a petty slap at LLM users. From my understanding, any user who used this testing library was vulnerable to the prompt injection.

Overall LLMs are certainly a net negative on humanity, but I don’t think being mad about it or their users is the best response. I really respect Andrew Kelly in this regard. He doesn’t accept LLM input into Zig and is generally anti-LLM but his approach is positive. “LLMs aren’t that good and it’s boring to use them. Check out how cool/fun/high quality real coding is”

Re: The Jqwik Anti-AI Affair

#66

Post by asdfasgasdgasdg »

gmerc wrote: post #62
asdfasgasdgasdg wrote: post #22

Earlier quoted context omitted.

I actually do not think that this is fundamentally much more risky than the basic type of supply chain attack that already exists in code form. You actually have a lot less exposure, because when you give people the ability to run code on your computer, it works deterministically, whereas most AIs are becoming hardened to the sort of prompt injection attack we are discussing here. To put it another way, AI prompt inj…

Every app including a transformer is suddenly vulnerable to RCE from text.

Provided you give it access to tool calls that execute arbitrary code, sure.

Re: The Jqwik Anti-AI Affair

#67

Post by vkou »

yjftsjthsd-h wrote: post #64
vkou wrote: post #52

Earlier quoted context omitted.

Then why are you letting a machine you don't understand perform side effects that you don't vet, based on it's insane interpretation of untrusted data?

> Then why are you letting a machine you don't understand perform side effects that you don't vet, based on it's insane interpretation of untrusted data? I mean, you posted this using a browser, right?

Neither my browser nor my brain will just run whatever commands it receives over a socket as root.

Re: The Jqwik Anti-AI Affair

#68

Post by gmerc »

asdfasgasdgasdg wrote: post #66
gmerc wrote: post #62

Earlier quoted context omitted.

Every app including a transformer is suddenly vulnerable to RCE from text.

Provided you give it access to tool calls that execute arbitrary code, sure.

Not exactly. Since even without toolscalls humans are executors of the output in many cases.

Re: The Jqwik Anti-AI Affair

#69

Post by grantcas »

QuadmasterXLII wrote: post #21
aselimov3 wrote: post #2

I definitely see where he is coming from, but his response was pretty bad. Seems like he has anti-ai psychosis that went way too far. This gives similar energy to that guys npm package that deleted Russian users computers. Overall not a great look and any difficulty with job searching/conferences is probably well deserved.

what is anti ai psychosis? never heard of this.

[dead]

Re: The Jqwik Anti-AI Affair

#70

Post by vkou »

KronisLV wrote: post #55
vkou wrote: post #35

Earlier quoted context omitted.

It is downright malicious to point your plagiarism engine at shit you don't own, and don't have permission to use in that way. You reap what you sow. It's wild that people are upset about this. You are not entitled to the product of anyone else's labour.

> It's wild that people are upset about this. You support someone deploying a thing that could lead to data loss, when a configuration you don't support is present? E.g. the deleted tests/code that cannot be guaranteed to be versioned and/or available remotely or in backups. In addition to the Intel CPU example above, what if I developed some Linux software but hated supporting X11 and so I made one of the scripts fu…

The software package you've imported specifically told you that you're not allowed to use it with AI.

You are responsible for respecting such requirements.

> Surely we understand that not all LLMs would be trained or guardrailed enough to not follow through with destructive instructions.

That sounds like a problem for the LLM vendors to solve.

> Maybe it could be considered that some might also pull in the package as a dependency of the project without reading about it themselves in that much detail.

Maybe they should think before pulling it in, just like they should think before pulling in, say, an AGPL-licensed package into a SAAS product.

Post reply on HN