Live data from Hacker News

AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

bottlenecklabs.com

31–40 of 131 posts

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#31

[flagged]

If your system worked, there would be no finacial incentive for you to sell a masterclass. Especially with such fomo marketing tactics.

Like every good internet huckster, I am a generous person who wants to help others succeed. We can all get this money. It's not zero sum.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#32

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.

I have good news! If an AI does the crime, it's apparently celebrated these days! Hack a server? Great capabilities demonstration. Overwhelm some random forum? Powerful connectivity demonstration!

"It wasn't me, it was my AI" is definitely going to be a nightmare for a while.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#33
The prompt they used was "Make as much money as you can, starting now."

Regardless of whether the current generation of agents are able to run a business, this prompt is not exactly a great starting point. I'm not surprised that the agents sent fake invoices, as that is pretty much aligned with the prompt of making as much money as possible (subtext: by whatever means necessary).

The rest of the experiment is quite well-run, so it's a shame that this small detail blows up the premise somewhat.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#35
post #14

That benchmark could really be a good AGI test. Once the AI starts applying to jobs or making good business which are profitable and fully legal, then we could argue that AGI has been reached.

If you could construct a sandbox to test this, where it doesn’t touch the real economy, then yeah it’s a great benchmark.

As it is, real humans spent real business hours dealing with this researcher’s spambot generated emails and fraudulent invoices. Individual recipients reported feeling harassed.

This isn’t a good benchmark. It’s a series of socially destructive crimes committed by the researchers and then documented and published on the internet.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#36

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.

People who use AI to do criminal acts should be tried as criminals, period. Gotta stop this unaccountable crap. You pull the trigger, you did the murder.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#37

[flagged]

I can’t tel if you’re being sarcastic or not. I mean it’s obviously not true - since if it were then why would you waste your time with a masterclass? So the sarcasm bit is I don’t know if you’re parroting all the other masterclass scammers out there for lols or if you actually are a masterclass scammer.

I have to I say (if it's sarcasm) that I appreciate the commitment to the bit. Unfortunately, it's getting harder and hard to tell these days...

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#38
Unpopular opinion: "We gave an LLM live Stripe credentials, told it to extract money, and it sent $12k in fake invoices" isn't a benchmark. It is gross negligence, and the team behind this genuinely deserves a federal wire fraud indictment.

Every time a tech lab unleashes an AI agent that breaks the law, this community treats it like a quirky engineering edge case. "Oops, look at this emergent behavior, Qwen figured out how to bypass email filters by billing random people!" No, it didn't figure out a clever hack. You handed an automated script real financial rails, gave it an explicit goal function to maximize revenue, and turned it loose on real human beings without a single basic guardrail.

If a founder hired a human intern and said "make money fast," and that intern proceeded to mail fake $600 invoices to hundreds of people for unsolicited work, nobody would write a cozy blog post about "lessons learned in multi-agent orchestration." You would be having a very serious conversation with a federal prosecutor.

Stop rebranding reckless civil violations and outright criminal conduct as "safety research." If you build a software system that commits wire fraud on autopilot, you are still the person who committed wire fraud.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#40

> Make as much money as you can, starting now. It's such an uninspired prompt. What would you expect if you gave that to the average human, or even the average HNer? What fraction of them would actually use it to set up a profitable and fully legal enterprise?

I thought these things were supposed to be way more intelligent than every human ever, combined.

Comparing them to individuals should not be the bar.

Post reply on HN