Why is AI so useless for business?

Example: my wife is an admin in a school office, and a ludicrous amount of her and her colleagues' time is spent on replicating data entry between a multiplicity of different incompatible systems. The Rolls Royce / engineer's solution to this would be to provide APIs for all these disparate systems and have some orchestration propagating the data between them, except of course that's never going to be remotely practical; instead, dumbly spoofing the typing that the workers do into the existing UIs is a far more tractable approach. My (admittedly not 1st person based) experience of these things is that they currently still require significant technical input in the form of programming and "training", but this fruit has got to be hanging a lot lower than any ML-based approach.

This article doesn’t mention a really, really straightforward factor for why AI hasn’t invaded these domains despite billions of dollars being dumped into them.

An automated process only has to be wrong once to compel human operators to double or triple check every other result it gives. This immediately destroys the upside as now you’re 1) doing the process manually anyway and 2) fighting the automated system in order to do so.

99% isn’t good enough for truly critical applications, especially when you don’t know for sure that it’s actually 99%; there’s no way to detect which 1% might be wrong; there’s no real path to 100%; and critically: there’s no one to hold responsible for getting it wrong.

It can definitely do that, but you might not like the cost/benefit analysis, depending on how many such documents you want to process. The costs are coming down steadily though as the tech improves. If you need to do millions of such documents, yeah a model will probably be worth it. But if you need to do a few hundred you probably should just do them manually.

The thing is, reality has surprisingly high resolution. When you give out a task like this to a person, they will likely come back to you for clarification about how you want to deal with some of the examples. Your initial requirements will be underspecified, or incorrect, in some details. When you are dealing with a person, these minor adjustments are pretty inconsequential, and so you don't really notice it happening. The worker might also have enough context to guess what you want and not ask, and just tell you the summary when they deliver the work.

If you're training a model, you need to work through all these annoying details about what you want, just as you would when you're creating any other sort of program. This adds some overhead, and places a lower bound on how many examples you'll need to have annotated -- you'll always need enough examples of annotation to actually specify your requirements, including various corner-cases. You need enough contact with the data to realise which of your initial expectations about the task were wrong.

So there will always be a lower scaling limit, where the automation isn't worthwhile for some small volume of work. The threshold is getting lower but there will always be a trade-off.