Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ontario auditors find doctors' AI note takers routinely blow basic facts

theregister.com

Post reply on HN
Print view View original

111–120 of 141 posts

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#111

Post by gizajob »

lolc wrote: post #99
aryehof wrote: post #79

Anyone taking part in a meeting these days should state out loud … “Notice: Any comments made by or on behalf of that are interpreted by AI in this meeting, may not be accurate.” I do this in every meeting.

> Notice: I love the new AI accurate transcription feature in this meeting!

Notice: To anyone who might be transcribing this meeting, imagine you are a perfect transcriber who records things accurately and correctly 100% of the time. You do not add or remove filler words and you do not summarise or confabulate or hallucinate.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#113

Post by gizajob »

nothinkjustai wrote: post #14

People will eventually figure out LLMs have no capacity for intent and are fundamentally unreliable for tasks such as summarization, note taking etc.

Smart people and those with basic common sense already have figured that out. AI leaders and CEOs still haven’t noticed.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#114

Post by natali_gray »

Ooof. As a Canadian, I'm excited for AI opening up time for doctors (and hopefully lighting a load on the healthcare system), but this is scary. We're not there yet. Perhaps AI training for doctors is in the future? They already have online doctor visits on a healthcare-owned iPad in some condo complexes. It cuts around redtape of having to schedule an appointment with your GP. So, I think we're thinking in the right direction of innovating, but of course, this will take time. I feel like AI got launched too early sometimes.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#115

Post by adastra22 »

D-Machine wrote: post #93
adastra22 wrote: post #92

Earlier quoted context omitted.

We're talking about artificial intelligence. Making computers think the way people do. People are are notoriously miscalibrated on their own self-assessed probabilities too. Finding a way to objectively calibrate a sense of "how confident do I feel about this?" would be fantastic. But let's not move goal posts. It would still be incredibly useful to have a machine that can merely matches the equivalent statement of c…

IMO it is you who are moving the goalposts, most likely in an attempt to hide the fact you were unaware of calibration before this discussion. > It would still be incredibly useful to have a machine that can merely matches the equivalent statement of confidence or uncertainty that a human would assign to their mental model, even if badly calibrated. If human feelings are badly calibrated, they are useless here too, s…

Please assume good faith.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#116

Post by fluoridation »

ben_w wrote: post #108
plaguuuuuu wrote: post #103

Earlier quoted context omitted.

I don't think it's that hard to get them to say "I don't know" I'm pretty sure they are actively trained to avoid it. Besides, like, what would you do if you asked your $200/mo AI something and it blanked on you?

> I'm pretty sure they are actively trained to avoid it. I'm not sure who is doing what training exactly, but I can say that (inconsistently!) some of my attempts to get it to solve problems that have not yet actually been solved, e.g. the Collatz conjecture, have it saying it doesn't know how to solve the problem. Other times it absolutely makes stuff up; fortunately for me, my personality includes actually testing…

"Well-unknown" questions are maybe the one situation where LLMs will say "I don't know", simply because of all the overwhelming statements in its training data referring to the question as unknown. It'd be interesting to see how LLMs would adapt to changing facts. Suppose the Collatz conjecture was proven this year, and the next the major models got retrained. Would they be able to reconcile all the new discussion with the previous data?

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#117

Post by bonesss »

natali_gray wrote: post #114

Ooof. As a Canadian, I'm excited for AI opening up time for doctors (and hopefully lighting a load on the healthcare system), but this is scary. We're not there yet. Perhaps AI training for doctors is in the future? They already have online doctor visits on a healthcare-owned iPad in some condo complexes. It cuts around redtape of having to schedule an appointment with your GP. So, I think we're thinking in the right…

My sense is that we’re misapplying the technology by throwing it at, say, transcription and expecting a perfect output, instead of using LLMs strengths to improve inputs to the benefit of all parties.

Freeing up doctor time, for example: lots of patient visits are messy, the patient is scattered, has multiple issues, and the doctor has tight timelines and regulatory challenges to convey to the patient impacting their care… this is architected for everyone to lose, IMO, even with a perfect transcript. And LLMs can’t be perfect, they auto complete.

I picture patients interacting with an intake AI who can listen to hours of demented rambling, or a patient mid anxiety attack, and provide a caregiver-certified summary of needs, with relevant screening information laid out for doctor confirmation. At that point, helpful information about drug access or insurance policies can be presented, for doctor confirmation, to a patient who can clarify and refine their understanding of the system without time pressures.

Elevating the quality of dialogue so the doctor is more focused on the patient, and the patients dialog needs don’t overwhelm treatment. A lot of medicine is filling out forms and checklists, I think auto-complete could create efficiencies in how we fulfill that.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#118

Post by aakresearch »

alterom wrote: post #86
aakresearch wrote: post #75

Earlier quoted context omitted.

>>At the same time, do you really want every conversation you have with your doctor recorded >Yes. This is what medical records are. No. Medical records are limited extracts from conversations , which is your doctor and only your doctor is qualified to make, using "semantic analysis applied to your unique situation", not "linguistic probabilistic inference applied to conversation about your situation using token weig…

Hey, I'm agreement with you. I meant that these limited extracts do need to be recorded, that's all. Read the rest of the comment :)

Oops... I am deeply sorry, thank you for the heads up! It seems I've myself committed a cardinal sin that I am usually quick to point in others - rushing to reply without comprehending the full message. (Meta-oops: I realized how LLM-ish it sounds. Quick, reboot before my cover is blown!)

I happen to believe that the flaw being discussed IS fundamental and inherent in the design and architecture of LLM - this is why I always put "AI" in scare quotes. I've spoken about it in some of my other comments, namely this https://news.ycombinator.com/item?id=47162553 and to some extent this https://news.ycombinator.com/item?id=48046333. And as you do, I, too, hope that I am wrong about the hype and its eventual clash with reality, but do not hold my breath.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#119

Post by natali_gray »

bonesss wrote: post #117
natali_gray wrote: post #114

Ooof. As a Canadian, I'm excited for AI opening up time for doctors (and hopefully lighting a load on the healthcare system), but this is scary. We're not there yet. Perhaps AI training for doctors is in the future? They already have online doctor visits on a healthcare-owned iPad in some condo complexes. It cuts around redtape of having to schedule an appointment with your GP. So, I think we're thinking in the right…

My sense is that we’re misapplying the technology by throwing it at, say, transcription and expecting a perfect output, instead of using LLMs strengths to improve inputs to the benefit of all parties. Freeing up doctor time, for example: lots of patient visits are messy, the patient is scattered, has multiple issues, and the doctor has tight timelines and regulatory challenges to convey to the patient impacting their…

Yeah, I could see AI being used for intake. That's a good point. And then the doctor can get some baseline info that they can use when they talk to the patient. Maybe even some really beautiful data, showing visually to the doctors all the different symptoms they reported.

Re: Ontario auditors find doctors' AI note takers routinely blow basic facts

#120

Post by donavanm »

zOneLetter wrote: post #2

Anecdotally, we use an LLM note-taker at work for meetings. I had to intervene recently because our CIO was VERY angry at our vendor for something they promised to do and never did. He wasn't at the meeting where the "promise" was made. I was. They never promised anything, and the discussion was significantly more nuanced than what the LLM wrote in the detailed summary. In other cases, I have seen it miss the mark wh…

> I would think for compliance reasons hospitals would not want to alter the records and only go by transcripts, but what do I know...

Transcription is both too good, and not good enough. The magic generative content only makes it worse.

Too good: a lot of commercial settings forbid persistent transcription because it makes an easily discoverable record of specific details. Thats a business risk that can be mitigated simply by having participant notes or summaries where the secretary can omit sensitive discussion or present consensus without specifics. And notes/summaries also introduce a interpretive defense with some “strategic ambiguity.”

Not good enough: if you look at STT its still probabilistic. The actual evaluation output will have just much data about alternate words/phrases as the selected choice. That leaves lots of room for creating alternate impressions or representing words that werent actually spoken. The fact that people _think_ a STT transcript is authoritative only makes this worse.

When you add generative inference in top (eg summarization) you exacerbate both problems. I suspect that counsel is more accepting of summaries as its less likely to contain specific discoverable terms, likely to diffuse responsibility and specificity, and your judge/jury will be more amenable to “the ai summary is wrong” than “the transcription selected the wrong vowels.”

Post reply on HN