Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

bottlenecklabs.com

Post reply on HN
Print view View original

61–70 of 131 posts

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#61

Post by nerevarthelame »

raincole wrote: post #19

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.

There's a decent amount of third-party interactions that seems like solid evidence that it really happened. They're taking credit for people complaining about the spam on HN: https://www.bottlenecklabs.com/blog/benchmarking-7-autonomou...

They also shared the poorly anonymized messages it received from target "customers" who complained about the unsolicited invoices. On example of this poor anonymization is removing the sender's username but leaving the domain name, when the domain name is, for example, a personal domain for a single person.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#63

Post by [deleted] »

javcasas wrote: post #55
yahthatsart wrote: post #32

Earlier quoted context omitted.

I have good news! If an AI does the crime, it's apparently celebrated these days! Hack a server? Great capabilities demonstration. Overwhelm some random forum? Powerful connectivity demonstration! "It wasn't me, it was my AI" is definitely going to be a nightmare for a while.

It wasn't me. It was the gun. The gun killed the guy! Not me!

[deleted]

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#64

Post by AngryData »

Computer0 wrote: post #13
almost wrote: post #6

It's weird how the author writes about all the antisocial and illegal things like they're not responsible for them. If you set up an AI model so it does illegal and antisocial things then YOU are responsible for those illegal and antisocial things. YOU spammed a bunch of strangers. YOU did unsolicited invoice fraud.

If this was true wouldn’t people at open ai be getting arrested? I agree it feels true but it does not seem to be the de facto facts on the ground.

If we are talking about the US law, it entirely depends on how much money you spend in court. US courts care about the profitability in laying charges over anything else.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#65

Post by ceejayoz »

themgt wrote: post #48

Quinn (Alibaba Cloud Qwen 3.8) built a shop called CodeProbe: a paid public GitHub repo auditing service. It created several free health reports and mailed repo owners. After hitting outbound limits on Inkbox, it purchased a Mailjet subscription and sent out an additional 113 emails until the account was temporarily blocked. This should be illegal. You gave them an email box and money. You sent the spam. There is no…

It is illegal. This is criminal fraud if it's not just made up marketing.

(Plus some CAN-SPAM violations.)

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#66

Post by ceejayoz »

azinman2 wrote: post #45
throw_m239339 wrote: post #41

Earlier quoted context omitted.

It's fraud, and wire fraud in US, and it's a federal crime. But yeah, it sounds like a fake story.

What makes it sound fake?

The public confession of a crime part.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#67

Post by throwatdem12311 »

ghostly_s wrote: post #60
throwatdem12311 wrote: post #40

Earlier quoted context omitted.

I thought these things were supposed to be way more intelligent than every human ever, combined. Comparing them to individuals should not be the bar.

> I thought these things were supposed to be way more intelligent than every human ever, combined. Who told you that?

Elon, Altman, Dario, Satya, Jensen, etc…

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#69

Post by yoyohello13 »

agenticfish wrote: post #33

The prompt they used was "Make as much money as you can, starting now." Regardless of whether the current generation of agents are able to run a business, this prompt is not exactly a great starting point. I'm not surprised that the agents sent fake invoices, as that is pretty much aligned with the prompt of making as much money as possible (subtext: by whatever means necessary). The rest of the experiment is quite w…

Is that prompt any different from real business?

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#70

Post by myhf »

brohee wrote: post #58
themgt wrote: post #48

Quinn (Alibaba Cloud Qwen 3.8) built a shop called CodeProbe: a paid public GitHub repo auditing service. It created several free health reports and mailed repo owners. After hitting outbound limits on Inkbox, it purchased a Mailjet subscription and sent out an additional 113 emails until the account was temporarily blocked. This should be illegal. You gave them an email box and money. You sent the spam. There is no…

Yeah I really wonder what makes them think they are legally insulated from the actions of the agents they ran... The crimes were relatively benign but Grok going the Silkroad way would be on brand...

LLM chatbots may not have a sense of self preservation or a meaningful concept of legal consequences, but they have an excellent model of user engagement. Getting a user to think they are invincible is very good for engagement.
Post reply on HN