"I just bought a 2024 Chevy Tahoe for $1"

> they are obligated to honor that "incorrect" price.

Clearly false. If the store owner sees the incorrect price, he can say "that's incorrect, it costs more... do you still want it?". If you call the cops, they'll say "fuck off, this is civil, leave me alone or I'll make up a charge to arrest you with". And if you sue, because the can of off-brand macaroni and hot dog snippets was mismarked the judge will award the other guy legal costs because you were filing frivolous lawsuits.

> "bots" can make legally binding trades on Wall Street, and have been for decades.

Both parties want the trades to go through. No one contests a trade... even if their bot screwed up and lost them money, even if the courts would agree to reverse or remedy it, then it shuts down bot trading which costs them even more than just eating the one-time screwup.

This isn't analogous. They don't want their chatbot to be able to make sales, not even good ones. So shutting that down doesn't concern them. It will be contested. And given that this wasn't the intent of the creator/operator of the chatbot, given that letting the "sale" stand wouldn't be conducive to business in general, that there's no real injury to remedy, that buyers are supposed to exercise some minimum amount of sense in their dealings and that they weren't relying on that promise and that if they were doing so caused them no harm...

The judge would likely excoriate any lawyer who brought that lawsuit to court. They tend not to put up with stupid shit.

I can assure you that, at least in the US, you can ask for a manager and start mentioning "attorney general" and you will get whatever price is on the cans of soup.

The hilarious part to me is the number of otherwise intelligent people concerned that this sort of stupidity is a threat to humanity.

The only real threat is from people willing to trust AI.

Sycophancy in LLMs is a real problem. Here's a paper from Anthropic talking about it:

https://arxiv.org/abs/2310.13548

I feel like a better use case for ChatGPT-like tools (at least in their current state) for customer support use cases is not actual live chat but more assisting companies in automating the responses to other non realtime channels for customer requests such as:

- email requests

- form based responses

- Jira/ZenDesk type support tickets

- forum questions

- wiki/faq entries

and having some actual live human in the mix to moderate/certify the responses before they go out.

So it'd be more about empowering the customer service teams to work at 10x speed than completely replacing them.

It'd actually be more equivalent to how programmers currently are using ChatGPT. ChatGPT is not generating live code on the fly for the end user. Programmers are just using ChatGPT so they aren't starting out with a blank sheet. And perhaps most importantly they are fully validating the full code base before deployment.

Putting ChatGPT-like interfaces directly in front of customers seems somewhat equivalent to throwing a new hire off the street in front of customers after a 5 minute training video.