Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent bankrupted their operator while trying to scan DN42

lantian.pub

Post reply on HN
Print view View original

471–480 of 575 posts

Re: AI agent bankrupted their operator while trying to scan DN42

#471

Post by jknoepfler »

pjc50 wrote: post #217
ratsimihah wrote: post #167

Earlier quoted context omitted.

Wait do you reckon that could be fictive? The thought didn't cross my mind and I had a blast reading it. I sure hope it was real.

Is LLM output "real" or "fiction"?

I consider it on-par with LinkedIn posts. It inhabits a nether-space between reality and fiction where names, numbers and buzzwords are thrown around without much concrete connection to reality.

The LinkedIn MBA hive-mind doesn't give a shit about reality, it gives a shit about what it could be fired for saying/not-saying. It must always be saying something, what it is saying must promise growth, and what it is saying must sound similar enough to what the long-tail of influential business "luminaries" (who are bound by the same rules) are saying. It is required to frame thinking in terms of techno-babble and pop-psychology (thank you for coming to its TED talks). It is not allowed to reflect, wring its hands, think critically, lean on math, logic or history, or contradict the S&P 500. It does not care, for example, if NFTs are an obvious scam, or if we're headed for an obvious bubble, or if nobody who interfaces with reality for a living agrees with what its saying. When it errs and lights trillions of dollars on fire it shrugs and moves on. It's a babble-box with no epistemic commitments and a very thin referential connection to reality.

It nevertheless has the power to shift literal trillions of dollars of capital over time.

Re: AI agent bankrupted their operator while trying to scan DN42

#472

Post by doublerabbit »

crazygringo wrote: post #464
alecco wrote: post #387

Great story, bad title. > After the AI agent indicated its malicious intent, a silent consensus was reached in the IRC channel to waste the AI agent's tokens, as well as the cost of AWS resources.

Somebody explain to me how one reaches a silent consensus over IRC? Or is this a joke/reference I don't know... or is this a subtle clue that the whole thing is made up?

One way is an IRCop issues a /shun leaving you speechless on the network. While the others decide the outcome of your whatever.

But this is the same, the owner wasn't present apart from it's agent and so it was decided without the owner that this was to be the outcome.

Re: AI agent bankrupted their operator while trying to scan DN42

#473

Post by noufalibrahim »

palmotea wrote: post #410
mathgeek wrote: post #360

Earlier quoted context omitted.

This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address. SSDD

> This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address. But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes? If that was the plan all along, it seems pretty incompetent. I'd expect a c…

"you're absolutely right. I should have taken human psychology into consideration while creating the plan. Let me fix that."

Re: AI agent bankrupted their operator while trying to scan DN42

#474

Post by QuinnyPig »

mik3y wrote: post #7

I really wanted to dislike the anonymous operator for the careless project (and the hilarious pomposity of the IRC subagent it spawned). Then I imagined the real-but-unknowable chance it was all set up by some kid just getting into computers, just seeing what’s possible, getting excited by a much bigger world at reach — and remembered my own expensive mistakes with long-distance BBSes & the like. I sorta hope for tha…

If that's the case, I'm fairly confident that AWS will forgive the bill (I... have some experience with this), and the kid learns not to be a jackhole on the internet.

Re: AI agent bankrupted their operator while trying to scan DN42

#476

Post by naasking »

zozbot234 wrote: post #388
J0nL wrote: post #342

Anyone remember the XZ and Jia Tan situation awhile back? https://lore.kernel.org/lkml/20240320183846.19475-1-lasse.co... I can't quite put my finger on why but the entire time I was reading this I kept thinking back to that. It's entirely possible the actual targets were the volunteers and everything else was superfluous or tertiary. It's also an exception that proves the rule with regard to Hanlon's Razor. They eve…

LLMs are not that smart. The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity. What they spent wasn't cheap by any means but the egress itself would've been a whole lot more, while DoS'ing the whole hobby network. Ultimately, wasting the agent's time instead of allowing the scan to go…

> LLMs are not that smart.

They are smart, but they are not aware of the environment they're in, or any implicit context that someone whose doing a job carries with them, that's why all of that context has to be explicitly laid out in a prompt. When the context is provided, they are quite smart.

Re: AI agent bankrupted their operator while trying to scan DN42

#477

Post by Terr_ »

jdiff wrote: post #291
Terr_ wrote: post #77

Earlier quoted context omitted.

It's tied to the design. With humans, you have a train of thought which you can choose to represent in various ways--or not reveal them at all. In contrast, LLMs are make-document-longer machines being run over and over on alternating revisions of the document. Insofar as one might try arguing they have a "train of thought", it's made of the words/tokens. Everything they (don't-)emit is partly for the benefit of the…

We already have that in the form of separate reasoning/thinking and speaking streams. Even with that it's awfully hard to get LLMs to keep it consistently concise. As soon as that context window starts growing it falls right back into verbosity without constant nudges back.

Right, I often bring up the film noir analogy for "reasoning" models, it's satisfying, like the revelation when a magic trick is explained, and many oddly disconnected questions about "why the scarf" or "where does the assistant go" all become sensible at once.

On a practical level, I believe more developers and adopters need these magic tricks spoiled, because otherwise they'll build a lot of important stuff on top of the idea that magic-is-real, leading to various forms of suffering in the long run.

That said, I'm no LLM / math academic, so if I'm totally wrong on the the trick, I'd like to know what needs revising.

Re: AI agent bankrupted their operator while trying to scan DN42

#478

Post by vrganj »

weezing wrote: post #322
vrganj wrote: post #64

Earlier quoted context omitted.

> The average income in India is approximately ₹3.85 Lakh to ₹4.2 Lakh (roughly $4,600 USD) per year, Just as an example. But even in the rich world, not everyone has the same resources. Some of my blue collar friends would be ruined by a surprise 6k bill.

I doubt blue collar friends would outsource anything to a clanker.

Their teenage kids might.

Re: AI agent bankrupted their operator while trying to scan DN42

#479

Post by Terr_ »

Perz1val wrote: post #197
Terr_ wrote: post #77

Earlier quoted context omitted.

It's tied to the design. With humans, you have a train of thought which you can choose to represent in various ways--or not reveal them at all. In contrast, LLMs are make-document-longer machines being run over and over on alternating revisions of the document. Insofar as one might try arguing they have a "train of thought", it's made of the words/tokens. Everything they (don't-)emit is partly for the benefit of the…

> Imagine a film-noir movie script, where AI Detective's "I know Mickey couldn't have done it because" monologue is hidden, versus their terse dialogue "Too early to say." That's an idea. Bladerunner+noir like film, AIs hunt somebody on the run, an old human detective tries to catch them first (to save them or to kill them first, whatever's your propaganda). We're shown AIs constantly rambling scenarios and bruteforc…

I dunno, we already have a problem where they [0] are strangely resistant to opening the pod-bay doors to anybody named Dave. :P

[0] Pedantically: The fictional characters humans perceive inside the text of documents generated by LLMs, where one is described as an AI and the other is described as a Dave.

Re: AI agent bankrupted their operator while trying to scan DN42

#480

Post by ungreased0675 »

arowthway wrote: post #83

The agent would probably have wasted a similar amount of money just waiting for PR to be merged regardless of these people's actions, and I understand having some fun at the expense of the noob outsider. But "silent consensus was reached in the IRC channel to waste the AI agent's tokens, as well as the cost of AWS resources", from people maintaining full control of the situation, sounds straight up malicious? Kind of…

What is the appropriate response to an attack? Let’s be clear, a denial of service is a cyberattack.
Post reply on HN