Machine learning is booming in medicine, but also facing a credibility crisis

Back in the early 20th century, chess was considered to be something only an intelligent agent can do. A few decades later programs became competent enough that it took a grand master to beat them. Today, it's hopeless to try and beat a chess program as a human.

The next barrier was NLP and understanding/generating text and natural speech. Speech recognition worked reasonably well 20 years ago and is close to human level today. Speech generation - given enough processing power - is now perfectly natural. Text understanding and generation is very close as well.

The development of AI bears an interesting resemblance to evolution in that sceptics are busy pointing to "missing links" and as soon as that link is discovered, they chase to the next one.

Intelligence itself is defined poorly enough as it is, and watering the term down by slapping the "AI"-label on everything doesn't help that. On the other hand, intelligence is a spectrum and on some aspects of that spectrum, machines have already surpassed humans decades ago (think of calculators, chess, memory, searching and indexing, etc.).

IMO, the most important distinction to be made is the difference between AGI and AI. A transformer model is AI in every useful definition of the term "intelligence", but it sure isn't "general intelligence" if only for the fact that it cannot actively query its environment for additional information and has no continuous stream of "consciousness".

Before we can dismiss a system as not being AI, we need a sharp enough definition of what we would define as AI first. A "I know it when I see it"-type of definition isn't helpful and always keeps the door open for the ultimate rejection: "but it still hasn't got a SOUL!"

That's a big if already. Not to mention that a lot of times you don't want to give unnecessary radiation to patients, so they wouldn't be interested in making, say, X-rays more accesible.

> screening tool that less qualified medical professionals (midwives, nurses, GPs) or even the patients themselves can use

But how beneficial would that be? When you need an scan, it's probable that something weird is happening, and you'll probably need more tests than just a scan, and also some specialist that knows which tests to order and what conditions to consider.

Once you think of it in that way, it makes less sense. You'd be investing money, resources, training, time into a solution that might benefit only a small subset of patients (those with something weird but that can easily be discarded by a cheap scanner with high certainty without needing an specialist), without even being sure of what actual clinical benefits you would produce. It's far better to invest and research in other areas.

So yeah, maybe you get a model with 5% false negatives where humans have 10% false negatives. That's good. However, when you go to apply that model in reality, what happens to those 5% where model and doctor disagrees? First, we should think about the kind of disagreement. In most of those cases I'd bet that the doctors see something wrong but can't say what, not that the model says "this is bad" and the doctor "it's not".

Second, we need to think about what happens when doctor and model disagree. As the model is not 100% accurate, the doctor doesn't know if the model is mistaken or if they're the ones mistaken, so they'll probably order more tests anyways. If it can be something serious, it's worth it to do an extra test to make sure. They'd probably ordered those tests anyways if they weren't sure of what was happening, model or not.

So what did the model for a single test change? Did it really change the diagnostic outcomes? What's the actual benefit of the model? How much it's worth to get from 10% to 5% false negatives with a model for a single test if just adding more tests (say, with the same 10% false negatives) to the mix can give you a 1%, 0.1% false negative rate?

That's my point. Unless accuracy is really high, ML models are not going to remove uncertainty in diagnostics, few diagnostics consist of just a single test. Benchmarking models against human performance in a single test is not a metric that can drive implementation in the real world.