I used Claude Code to get a second opinion on my MRI

I work at a telemedicine company. We’ve benchmarked a few frontier LLMs on public medical imaging datasets. One test included high-quality and high-consensus otoscopic images. We didn’t anticipate the models to do well on something so niche, but what concerned us was how poorly calibrated the models were.

I know you can’t trust an LLM’s self-assessed “confidence” of a prediction, but I’ve found that confidence can at least be directionally correct for some tasks. For our benchmarks, however, confidence was poorly correlated. What’s worse is that binary classification models (“Do you see $diagnosis in this photo?”) highly influenced the LLM to confidently predict $diagnosis.

I’m concerned for those using LLMs for diagnostics, and getting confidently led to the wrong conclusion.

Personally my favourite feature of the new ai world is not when I use it directly but it's when one of my managers uses it to try to fix a problem, then issue to me their findings and I have to defend my process to someone who understands neither my process, their suggested solution nor often the problem they're solving in the first place.

This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief.

It is weirdly religious in a way, because if you were to present contrary evidence (e.g. experts in a field weighing in about how plausible sounding responses are bunk), you would only be told you don’t believe enough in the long term potential and capabilities.

Don’t get me wrong, I think we all agree capabilities will eventually improve (and farther-future capabilities could reasonably surpass experts), but really is unclear if the current transformer architectures with their probabilistic/hallucinatory outputs will plateau before they surpass current experts abilities in all promised fields.

No, this flow is actually very good.

Like any domain, when you have questions or need a solution, you make research first, then you ask a specialist.

If you explain well the symptoms and context you can have proper advices and then decide on the path next:

    Case A) It looks benign and advices / information that you collected seem reasonable, then you go your way.

    Case B) You need second opinion of a specialist because the subject is too complex, or there are medications that you need approval.

Once you have challenged LLMs, and read about the topics over and over then you genuinely become really good at understanding it (especially if you triangulate over LLMs and ask them to challenge, you start to have genuine questions). No matter if the answer is right or wrong, you have elements. Maybe you missed the point, but you come prepared.

At home you have the time to assess the options, pros and cons of each approaches, the possible questions to ask and then challenge the doctor.

Shared decision-making is an actual evidence-based model of care, and patients who arrive understanding their condition and carrying specific questions tend to get better attention and better outcomes.

Some doctors get annoyed, because they have big ego and choose to be patronizing, but it is exactly their job to answer such questions.

    With LLMs, it's quite good, you get nuanced and rather useful answers.

    Before LLMs, no matter the topic you searched for, the answer was the same: "you have cancer / an [obviously deadly] rare disease"

The other problem, in many places:

    • The doctors are not affordable
    • They are too busy for you (

+ you can have all of these factors together.

So, you have something deeply bothering you, your only appointment is in 4 months. It would be insane not to take the time to explore different solutions and not to come informed about the topic.

If you express your prompt properly and do not rely on imagery, you can absolutely have top-tier advices.