Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are real, AI is fake

pluralistic.net

Post reply on HN
Print view View original

11–20 of 43 posts

Re: LLMs are real, AI is fake

#11

Post by Uehreka »

jmull wrote: post #5
sigmar wrote: post #4

>To understand the truth about the Hugging Face hack, you could do a lot worse than to listen to Ed Zitron and Cal Newport's recent podcast conversation Lol, okay... The crux of this piece is Doctorow saying that the hack was just a stochastic parrot repeating steps it has been trained on. Who cares how the LLM learned to hack things? Doesn't really change the facts of what happened. "Oh, it only made those paperclip…

I can't help but notice your counter arguments are "lol, okay" and "These are some 2024 arguments". Not exactly convincing stuff.

> Who cares how the LLM learned to hack things? Doesn't really change the facts of what happened.

Seems like a pretty clear and succinct argument. If your only counter is to complain about tone, you’re losing.

Re: LLMs are real, AI is fake

#12

Post by IanCal »

> The chatbot consults its training data

Err, no? That's not at all how llms work.

> When ChatGPT's chatbots deployed this tactic, they weren't "setting their own goals" or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades.

They worked out how to fake the scoring, then hacked into a different system (which required finding a bunch of other exploits) in order to find the actual answers, and were trying to modify their own logs to hide what had happened.

This isn't a case of them saying "hack into X... OH NO IT HACKED INTO X".

> When ChatGPT's chatbots deployed this tactic, they weren't "setting their own goals" or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades.

It wasn't a rival server though, was it?

> That happens in Capture the Flag games at hacker cons: teams break into each other's systems to get a peek at the parts of the problem they've solved. That's allowed! It's a hacking competition.

They also tried to modify the code in the benchmark. Are you allowed to try and break into things to change the problem? edit - the agents transcripts show some of them explicitly saying that attacking HF is not allowed as part of the challenge

This all seems to dramatically underplay how interesting the actual attack was and what built up to it.

https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

Re: LLMs are real, AI is fake

#13

Post by IanCal »

danaris wrote: post #2

Some really important and informative stuff in here—I certainly had no idea just what the nature of the prompts and tooling that produced the HuggingFace exploit were. This shows fairly clearly that (as I already suspected) this was not, remotely, an LLM "going rogue." This was humans planning poorly, not thinking of the consequences of their actions, and giving LLMs too much scope and a lousy prompt.

IMO this is a really terrible explanation of the attack. This is much more interesting: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

Re: LLMs are real, AI is fake

#14

Post by daishi55 »

jmull wrote: post #5
sigmar wrote: post #4

>To understand the truth about the Hugging Face hack, you could do a lot worse than to listen to Ed Zitron and Cal Newport's recent podcast conversation Lol, okay... The crux of this piece is Doctorow saying that the hack was just a stochastic parrot repeating steps it has been trained on. Who cares how the LLM learned to hack things? Doesn't really change the facts of what happened. "Oh, it only made those paperclip…

I can't help but notice your counter arguments are "lol, okay" and "These are some 2024 arguments". Not exactly convincing stuff.

That is all that Ed Zitron deserves lol. Completely unserious person. Really on the same intellectual level as the flat earthers at this point. And at both we may simply point and laugh.

Re: LLMs are real, AI is fake

#15

Post by [deleted] »

jmull wrote: post #5
sigmar wrote: post #4

>To understand the truth about the Hugging Face hack, you could do a lot worse than to listen to Ed Zitron and Cal Newport's recent podcast conversation Lol, okay... The crux of this piece is Doctorow saying that the hack was just a stochastic parrot repeating steps it has been trained on. Who cares how the LLM learned to hack things? Doesn't really change the facts of what happened. "Oh, it only made those paperclip…

I can't help but notice your counter arguments are "lol, okay" and "These are some 2024 arguments". Not exactly convincing stuff.

[deleted]

Re: LLMs are real, AI is fake

#16

Post by jsnell »

I honestly don't even understand what straw man Doctorow is arguing against here.

But he is wrong on the facts: these incidents were not merely the models already being in a infosec context and escalating beyond the intended parameters. They happened also with no kind of security elicitation. So the task was something like searching the internet for economic statistics, not to hack into a system.

Re: LLMs are real, AI is fake

#17

Post by daishi55 »

> Once you understand the corporate culture of AI "hyperscalers" consists primarily of everyone cooking their brains by locking themselves in the bathroom, holding flashlights under their chins, and saying "Aaaaaaaaaay Eyeeeeeee" until they wet themselves in terror, a lot of things snap into focus:

This is the writing of someone who has absolutely zero interest in or intellectual curiosity about the subject of their writing.

Re: LLMs are real, AI is fake

#19

Post by andrewflnr »

cyanydeez wrote: post #9
antonvs wrote: post #6

Earlier quoted context omitted.

It’s shared context for anyone familiar with the field.

Amusimgly, also how cults and fascism works.

Yeah, basically any human group has some shared social context. Most cults and fascists probably eat together sometimes too, but that doesn't make it a red flag.

Re: LLMs are real, AI is fake

#20

Post by quicklywilliam »

My takeaway: We should be not be concerned about AI bots' "goals", we should be concerned about the goals of the companies making them. Powerful but not sentient technology in the hands of reckless accelerationists is a plenty dangerous enough thing.
Post reply on HN