Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

bottlenecklabs.com

Post reply on HN
Print view View original

111–120 of 131 posts

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#111

Post by watwut »

echelon wrote: post #94
ceejayoz wrote: post #65

Earlier quoted context omitted.

It is illegal. This is criminal fraud if it's not just made up marketing. (Plus some CAN-SPAM violations.)

> It is illegal. Probably not forever. Eventually the models will be good enough for this to work. And it will work. Think about it: in the limit, the agents won't be emailing people in the future, they'll be directly contacting one another to do business and trade. Every new data center is an inch further towards the automation of value creation, and that includes outbound sales and business process automation. I'm…

There was no value creation involved here. It was a fraud, plain and simple. If you did this manually, it would still ve a fraud.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#112

Post by c22 »

Tade0 wrote: post #51
ElProlactin wrote: post #17

[flagged]

There's an interesting phenomenon I've observed: I am an extremely naive person in all aspects of life, except for money. I was genuinely with you until I started reading that last paragraph.

The last paragraph is the punchline.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#114

Post by echelon »

watwut wrote: post #111
echelon wrote: post #94

Earlier quoted context omitted.

> It is illegal. Probably not forever. Eventually the models will be good enough for this to work. And it will work. Think about it: in the limit, the agents won't be emailing people in the future, they'll be directly contacting one another to do business and trade. Every new data center is an inch further towards the automation of value creation, and that includes outbound sales and business process automation. I'm…

There was no value creation involved here. It was a fraud, plain and simple. If you did this manually, it would still ve a fraud.

There are lots of "not clearly legal" things that turn into big business.

- YouTube had dubious legality when it started and definitely benefited from lax copyright enforcement initially

- PayPal didn't have all the licenses it needed to transfer money between states

- Spotify used pirated music when it started

- Uber and Lyft broke rules around taxis

- Square captured magstripe data over an analog port, in violation of every credit card rule (Jack Dorsey's "break the rules" mantra). He tells each of his employees this story when he onboards them.

- Companies scraping data to train models

- ElevenLabs growing big off of deepfake celebrity audio

...

A lot of new markets start out by totally and completely breaking the norms.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#115

Post by KennyBlanken »

utopiah wrote: post #84
themgt wrote: post #48

Quinn (Alibaba Cloud Qwen 3.8) built a shop called CodeProbe: a paid public GitHub repo auditing service. It created several free health reports and mailed repo owners. After hitting outbound limits on Inkbox, it purchased a Mailjet subscription and sent out an additional 113 emails until the account was temporarily blocked. This should be illegal. You gave them an email box and money. You sent the spam. There is no…

Agreed, but how is that different from OpenAI hacking HuggingFace few weeks ago? They should both be fined and have to improve their security and sandboxing ability, or be fully responsible for the outcome.

....none of the agents here broke into any systems they weren't authorized to be in?

> should be fined

Weird way of spelling "criminally charged."

> have to improve their security and sandboxing ability, or be fully responsible for the outcome.

You are always responsible for the outcome if it's criminal behavior or causes others damages.

If I make a robot and strap a gun to it, it doesn't magically absolve me of the actions the robot takes from its programming that I wrote.

And before someone says "but this an LLM!"...yeah, which is still programming and data. And being non-deterministic doesn't help your case...it hurts it.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#116

Post by KennyBlanken »

karmakurtisaani wrote: post #47
yahthatsart wrote: post #32

Earlier quoted context omitted.

I have good news! If an AI does the crime, it's apparently celebrated these days! Hack a server? Great capabilities demonstration. Overwhelm some random forum? Powerful connectivity demonstration! "It wasn't me, it was my AI" is definitely going to be a nightmare for a while.

And who could forget the classic "let's train our model with stolen material"-scheme!

"Let's steal the voice of one of the western worlds' most famous female actors, after she explicitly told us no!"

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#117

Post by Brian_K_White »

throwaway_7274 wrote: post #88
Brian_K_White wrote: post #46

Earlier quoted context omitted.

But if you give any more specific direction, then the result is partly the result of your input, not the ai. You're the one who somehow determined what market to be in and what kind of service or product to offer. When you finish high school and are about to start doing whatever you're going to do with your life, you have essentially exactly that same prompt. The rest of the world doesn't tell you what to do and then…

Only psychopaths interpret the prompt given to them by the world as, "make as much money as you can."

Only pedants interpret a detail as the essence.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#118

Post by KennyBlanken »

javcasas wrote: post #95
prasadjoglekar wrote: post #79

Earlier quoted context omitted.

To be fair - no one is suggesting the gunslinger shouldn't go to jail or worse. It's using that crime as an excuse to take other people's guns away. In this example, the equivalent would be everyone gets their AI models banned vs. putting the perpetrator of this fraud in jail.

It wasn't me. It was the car. The car ran over and killed the guy! Not me! We have regulations and requirements and licences and insurance in most of the civilized world for dangerous stuff that is intended for public use. Maybe AI should also have some of that.

> It wasn't me. It was the car. The car ran over and killed the guy! Not me!

I know you were referencing AI, but this problem predates AI by decades.

Go open up any news website, newspaper, or turn on the TV.

"Man struck by speeding car and killed"

"Vehicle plows into pizza shop"

"Truck takes out telephone pole"

...and these days even bystanders will blur plates in the photos they post. The police won't release any info about the driver, the news won't either.

Listen to friends, family, coworkers talk.

"I can't believe that car just ran that red light!" "The other day I was almost hit by a car in the crosswalk." "All those cars honking their horns late at night are keeping me awake."

Etc.

Once you see it, you can't unsee just how thoroughly the auto industry has managed to transfer perception of responsibility from the operator to the object.

Re: AI models ran real businesses: They sent $12,431 in fake invoices, lost $3,200

#120

Post by falcor84 »

Brian_K_White wrote: post #117
throwaway_7274 wrote: post #88

Earlier quoted context omitted.

Only psychopaths interpret the prompt given to them by the world as, "make as much money as you can."

Only pedants interpret a detail as the essence.

Which part of that prompt is detail, and which the essence?
Post reply on HN