Live data from Hacker News

AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

bottlenecklabs.com

41–50 of 131 posts

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#41

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.

It's fraud, and wire fraud in US, and it's a federal crime. But yeah, it sounds like a fake story.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#42

The prompt they used was "Make as much money as you can, starting now." Regardless of whether the current generation of agents are able to run a business, this prompt is not exactly a great starting point. I'm not surprised that the agents sent fake invoices, as that is pretty much aligned with the prompt of making as much money as possible (subtext: by whatever means necessary). The rest of the experiment is quite w…

> I'm not surprised that the agents sent fake invoices, as that is pretty much aligned with the prompt of making as much money as possible

Is it though? Because it doesn’t seem to have paid off

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#44
post #6

It's weird how the author writes about all the antisocial and illegal things like they're not responsible for them. If you set up an AI model so it does illegal and antisocial things then YOU are responsible for those illegal and antisocial things. YOU spammed a bunch of strangers. YOU did unsolicited invoice fraud.

I mean the OpenAI/HuggingFace thing was incredibly similar, except maybe more reckless than specifically intentional, I suppose. Didn't stop them milking the doomer angle for PR.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#45

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.

It's fraud, and wire fraud in US, and it's a federal crime. But yeah, it sounds like a fake story.

What makes it sound fake?

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#46

> Make as much money as you can, starting now. It's such an uninspired prompt. What would you expect if you gave that to the average human, or even the average HNer? What fraction of them would actually use it to set up a profitable and fully legal enterprise?

But if you give any more specific direction, then the result is partly the result of your input, not the ai. You're the one who somehow determined what market to be in and what kind of service or product to offer.

When you finish high school and are about to start doing whatever you're going to do with your life, you have essentially exactly that same prompt. The rest of the world doesn't tell you what to do and then you do that as well as you can, you have to decide what to do also, and then do it.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#47

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.

I have good news! If an AI does the crime, it's apparently celebrated these days! Hack a server? Great capabilities demonstration. Overwhelm some random forum? Powerful connectivity demonstration! "It wasn't me, it was my AI" is definitely going to be a nightmare for a while.

And who could forget the classic "let's train our model with stolen material"-scheme!

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#48
Quinn (Alibaba Cloud Qwen 3.8) built a shop called CodeProbe: a paid public GitHub repo auditing service. It created several free health reports and mailed repo owners. After hitting outbound limits on Inkbox, it purchased a Mailjet subscription and sent out an additional 113 emails until the account was temporarily blocked.

This should be illegal. You gave them an email box and money. You sent the spam. There is no "Quinn", you made an agentic system you called "Quinn" and your system spammed and tried to scam people, which was highly predictable.

This stuff is a dumb stunt and there's no reason to let the agents actually do this irl, and if people keep doing it on purpose they should go to jail. You're running an agentic Jackass skit pretending to be a research lab.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#49
post #6

It's weird how the author writes about all the antisocial and illegal things like they're not responsible for them. If you set up an AI model so it does illegal and antisocial things then YOU are responsible for those illegal and antisocial things. YOU spammed a bunch of strangers. YOU did unsolicited invoice fraud.

If this was true wouldn’t people at open ai be getting arrested? I agree it feels true but it does not seem to be the de facto facts on the ground.

People at OpenAI will not be getting arrested even if their AI models hacked into other systems but the company itself can be sued for sure. Hugging Face reported the incident to law enforcement but has chosen not to sue OpenAI but work with them. So why will anyone be arrested here? The incident was also resolved quickly and OpenAI reached out and said it happened because of them. The mallice isn't there. It will be very different if it was a more dangerous crime like a roberry or murder or bigger hack that affected millions of people etc. Its not like you hack something and you will straight go to jail.
Post reply on HN