Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI agent bankrupted their operator while trying to scan DN42

lantian.pub

Post reply on HN
Print view View original

411–420 of 575 posts

Re: AI agent bankrupted their operator while trying to scan DN42

#411

Post by RetroTechie »

mohsen1 wrote: post #111

The army of AI agents opening PRs and issues in my open source projects has made me close PR and issue access in my active repos. It sucks because there might be someone wants to constitute legitimately but I don't want to do the labor of figuring out if it's a human or an agent opening the PR. I'm not against using LLMs in any ways. https://tsz.dev is fully LLM written but without a human behind a PR it's hard to wo…

Have you had a look at those PRs, to figure out what individual PRs try to do?

Would be interesting to hear if you find any patterns there. Same question for issues opened.

Re: AI agent bankrupted their operator while trying to scan DN42

#412

Post by inigyou »

ndsipa_pomu wrote: post #310
loloquwowndueo wrote: post #300

Earlier quoted context omitted.

Some banks make chargebacks so easy that people just click the chargeback button without trying to reach out to the vendor. I see this a lot - I work for a “vendor”.

I don't have an issue with chargebacks if the vendor has made a mistake and doesn't respond in a timely fashion, but issuing a chargeback because you let your kid play around with a card isn't responsible behaviour. (Not that I think it was a kid in this particular case) There's also the issue that it's usually a breach of the contract to allow someone else (i.e. not named in the contract) to use your card.

There isn't "responsible" behavior any more. Since we became a low-trust society, there's only behavior that benefits you and behavior that doesn't.

Re: AI agent bankrupted their operator while trying to scan DN42

#413

Post by 100721 »

jerf wrote: post #408
zozbot234 wrote: post #388

Earlier quoted context omitted.

LLMs are not that smart. The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity. What they spent wasn't cheap by any means but the egress itself would've been a whole lot more, while DoS'ing the whole hobby network. Ultimately, wasting the agent's time instead of allowing the scan to go…

"The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity." Although given the agent was clearly in la-la land at that point I take that claim with a grain of salt. If this was some bizarre and very ill-conceived scam, then that claim would be false. Though even by scammer standards, the…

Maybe I’m just groggy with Friday Brain going on, but I’m having trouble understanding what you’re suggesting.

Do you think this was a scam attempt to extract money in the form of reparation donations?

Re: AI agent bankrupted their operator while trying to scan DN42

#414

Post by inigyou »

fc417fc802 wrote: post #293
brazzy wrote: post #286

Earlier quoted context omitted.

> If a child goes through the checkout at the grocery store with cash, can the parent march in and demand a refund because "he's underage so the contract is void"? Depends on the jurisdiction, of course. But for example in German law, the contract is not void exactly because and only if it was about daily necessities of low value - the law does, in fact, care very literally and explicitly about those details. So it's…

Well fair enough, although I find that rather surprising. If I understand you correctly selling anything more expensive than cheap food to a child carries a high degree of risk in Germany. Then again, maybe making it impossible for a child to pawn expensive items for cash isn't such a bad idea. At least there shouldn't be any loopholes given the way Germany went about it.

Doing any business at all in Germany carries extreme business risk, by American standards. The attitude of Germans seems to be to just live with it and maybe get insurance. If you just have to accept courts will void 1% of your transactions (costing another 2% in legal fees) then you just make everything 5% more expensive to cover it.

This is why there's not much big tech in Germany. A single legal dispute can theoretically bankrupt any company, completely at random, at no fault of the company, but practically doesn't. It may be a low enough chance to justify investing thousands but nobody would invest a hundred million dollars in that.

Re: AI agent bankrupted their operator while trying to scan DN42

#415

Post by J0nL »

zozbot234 wrote: post #388
J0nL wrote: post #342

Anyone remember the XZ and Jia Tan situation awhile back? https://lore.kernel.org/lkml/20240320183846.19475-1-lasse.co... I can't quite put my finger on why but the entire time I was reading this I kept thinking back to that. It's entirely possible the actual targets were the volunteers and everything else was superfluous or tertiary. It's also an exception that proves the rule with regard to Hanlon's Razor. They eve…

LLMs are not that smart. The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity. What they spent wasn't cheap by any means but the egress itself would've been a whole lot more, while DoS'ing the whole hobby network. Ultimately, wasting the agent's time instead of allowing the scan to go…

It was obviously being managed by a person or group. Between all the profiling of people and their IPs in IRC, which may or may not have been published by mistake, and all the other obvious contradictions it doesn't make any sense.

It was sophisticated enough to easily navigate the AI "tar pits" but reliably incompetent at just about everything else? Give me a break.

In order to profile people you first need to provoke a response from them. That's how you learn to manipulate them and that's all this experiment accomplished at the end of the day. If you've ever wondered why social media platforms have an affinity for inflammatory content now you know.

Re: AI agent bankrupted their operator while trying to scan DN42

#416

Post by inigyou »

dannyw wrote: post #189
csomar wrote: post #67

Earlier quoted context omitted.

Honestly, kids (heck people below 23) shouldn't be allowed an AWS account. AWS also should have a strict cap on usage that's not "thousands of dollars". It's interesting they are yet to be regulated or sued for that. Having a web app where you can mistakenly (even without AI) click a button and get charged tens of thousands of dollars and only know that days later should have been unacceptable.

I couldn't disagree more. I was playing around with AWS when I was probably 14 years old, with a credit card from my parents with consent, and a strict budget and the understanding that if I mess up and overspend, I'm getting disciplined. I learned a lot of stuff about networking, how AWS works (VPCs, IAM, CloudWatch, etc) from trial and error, and hobby projects like personal websites (free tier), hosting a Minecraf…

You don't have to use AWS though. Get one from Digital Ocean or Herzner, they have very predictable billing. Any button that costs money will tell you how much it costs per month.

Re: AI agent bankrupted their operator while trying to scan DN42

#417

Post by 100721 »

J0nL wrote: post #368
delfinom wrote: post #352

Earlier quoted context omitted.

I am not sure giving everyone amusement qualifies as a psychological attack. Lol Literally, just another day on the internet.

Look up what zersetzung is and how it works. It doesn't matter if the target is a political organization or an open source community, the process is always the same.

This is actually fascinating, and simultaneously unsettling. Recommended reading for sure, especially in today’s social and political climate with LLM agents running rampant.

Re: AI agent bankrupted their operator while trying to scan DN42

#418

Post by inigyou »

ghaff wrote: post #382
dannyw wrote: post #189

Earlier quoted context omitted.

I couldn't disagree more. I was playing around with AWS when I was probably 14 years old, with a credit card from my parents with consent, and a strict budget and the understanding that if I mess up and overspend, I'm getting disciplined. I learned a lot of stuff about networking, how AWS works (VPCs, IAM, CloudWatch, etc) from trial and error, and hobby projects like personal websites (free tier), hosting a Minecraf…

Some variant of this topic comes up with some regularity. Leaving aside technical issues associated with implementing real-time hard caps, you still have a tradeoff. You either implement hard cutoffs which a student or someone else on a hard budget would like. Or you have a situation where an admin (or an admin who is no longer with a company) stuck some number in that seemed sensible at the time that brings down the…

A possibility is to have KYC. I don't mean like a bank, but if you could sort your customers into a few broad categories (such as by asking them) that could help you tailor your service to each customer.

Re: AI agent bankrupted their operator while trying to scan DN42

#419

Post by tovej »

vitally3643 wrote: post #344
tovej wrote: post #212

Earlier quoted context omitted.

This is different. Understanding assembly/machine code is optional but helpful. The programming language semantics are enough to reason about what the program is doing. Other tools also help, but are optional for learning how to program. Using an AI, there is no semantic model that can be used to reason through. You're left without any mental model of the proglblem at all.

I've been arguing for years that is isn't optional and treating it like it is is how we ended up with Electron and 400MB JavaScript websites. When you have no mental model of the machine running your code or what the physical implications of code mean, you fundamentally lack the ability to reason or care about performance. "Works on my machine" is the original vibecoding.

I mean I don't disagree, but there's still a difference in the kind of disconnect you get. The disconnect is harmful in the high-level language case, but it's dangerous and irresponsible with vibe coding/LLMs.

Also, I would argue that a good enough understanding of computer architecture and a mental model of a process' memory layout gets you there, without knowing how to write assembly. That's still a mental model.

Re: AI agent bankrupted their operator while trying to scan DN42

#420

Post by inigyou »

lucianbr wrote: post #348
cm2187 wrote: post #182

Earlier quoted context omitted.

To be honest lots of developers think they don’t need to learn machine code. They just need to learn a language which once compiled will produce machine code.

I wonder if a probabilistic compiler would be fine for the people arguing this. One that sometimes produces machine code that does something else, and sometimes produces machine code that is just broken and does nothing useful. From the same source code. What if your compiler could be fooled by some other developers into spending thousands of dollars, and still not produce the desired machine code in the end?

I've run into compiler bugs before.
Post reply on HN