Why is AI so useless for business?

> instead of having the work reviewed by a second person, you can have model + person

But the argument still applies then - if only one incident occurs where a catastrophic mistake was not spotted by the model failsafe, and if later investigations show that the mistake could've been easily spotted by a second human, a human will be installed as a failsafe for the failsafe.

I think the general problem is the following: if a human makes a grand mistake, it can usually be attributed to a temporal lack of care, or just random bad luck, or happened because the person was having a bad day, or... it is also generally understood that making a grand mistake will be such a shock for the person responsible that this person will most likely never make the same mistake again. On the other hand, if a machine makes a grand mistake, the intuition of the general public, trained by centuries of experience with techology, is that this machine will make the same mistake again and again and again, when prompted with the same input. If the model is not designed to learn from mistakes in production, this will of course actually be true.

> 99% isn’t good enough for truly critical applications, especially when you don’t know for sure that it’s actually 99%; there’s no way to detect which 1% might be wrong; there’s no real path to 100%; and critically: there’s no one to hold responsible for getting it wrong.

Like with self-driving cars?

One thing I haven't seen explicitly mentioned is the (probably) intrinsic limitations of AI/ML in classifying/predicting human behavior. For a number of years I have worked on fraud prevention tasks where the goal was to take in some information about a payment and decide whether it was fraudulent or not.

Even though what we were doing was primitive and could have benefited from a lot more ML, I suspect that even then the best that you could have gotten would have been some sort of anomaly detection system that can catch a good share of the kind of fraud that you have seen in the past, but will never be very good at detecting an intrinsic change in fraudster behavior.

On top of this, especially when dealing with humans, you often are expected to be able to explain why a certain decision was made. Setting payments aside, think of predictive policing or sentencing decisions. In those cases ML is essentially guaranteed to build in all sorts of biases regarding somebody's race, gender, place of living etc.

But isn't also the case that humans understand the consequences and the depth of multi-variable decisions?

For example, Amazon Seller Central, Google, Youtube, can "outsource" their customer service to AI, because they are pretty much the only players, so customers have to suck it and deal with the frustration of a terrible experience by not getting help and not talking with a human.

With any other business if they get automated replies that don't solve a customer issue and they are unable to talk with a human, 99% of them say "fuck it" and go somewhere else.

This is one small realm of the relationship of AI and businesses. Then you have employees, suppliers, supply chains, finances, internal processes, so many multi-variable, fragile and nuanced systems that I doubt if they're not developed in-house, they'll probably do more damages than solve problems.

Yep that's a great example. For other businesses:

Pharmacy - adding 1% more of a chemical compound to a pill could be lethal, but not otherwise detected for 3-12 months (pills sit on shelves for a long time). Figuring out why that happened will be very difficult.

Manufacturing - molding plastic parts requires very specific mixes of chemicals. Too much of one and it becomes brittle under certain temperatures, too little and it warps down under the warranty period. So if you manufacture the mold of a baby car seat, and you add or subtract 1% of a chemical, that car seat could break / shatter when it's involved in a collision. Terribly large lawsuits would occur because some C-level someone adopted a process to save headcount and reduce oversight.

Business processes are a lot like software - the first 80% takes 20% of the time and the remaining 20% takes 80% of the time.