LLMD: A Large Language Model for Interpreting Longitudinal Medical Records

Hang on -- while this is a cool result, beating a limited number of models that you chose to include in your comparison does not qualify LLMD-8B as SOTA. (For example, Claude 3 Sonnet scores 10 percentage points higher.)

>This result confirms the power of continued pretraining and suggests that records themselves have content useful for improving benchmark performance.

In support of this conclusion, it would be informative to include an ablation study, e.g. evaluating a continued pre-training data set of the same total size but omitting medical record content from the data mix.

Have you checked out dataset building with nemotron? The nemotron synthetic data builder is quite powerful.

Moreso, check out model merging. It's possible if you merge some of your model against llama3.1 base it may perform much better.

Check out max labonne's work on hugging face

I can't understand your methods without example prompts or code, so it's hard for me to interpret the data in figure 6. It will be important to document the methodology carefully to avoid concerns that your "text response" methodology is unfairly punishing other models.

In any case, since the methodology that Anthropic applied is documented and straightforward, it would be possible to do an apples to apples comparison with your model.

(I'm also very curious to know how 3.5 Sonnet performs.)

Is your text methodology based on CoT (like the "PubMedQA training dataset enriched with CoT" you trained on) or a forced single token completion like Anthropic used in their evaluation? In the latter case, I'm not sure how "text responses" differ from log probabilities at Temperature T=0 (i.e., isn't the most likely token always going to be the text response?)

So for method: We did NOT force single token responses. Our goal was to say "if we use [model x] to serve an app for [this task], how accurate would it be?" -- so we wanted to get as close to pasting the prompt directly in and just grading if the output was correct or not. In some cases, that directly works; in others, we'd have to lightly adjust the system prompt (e.g. "Answer ONLY yes, no, or maybe"); and in some cases, it required significant effort (e.g. to parse stubbornly verbose responses).

For the models like GPT-4o, Llama3-70B, and Sonnet that have great instruction following behavior, this works in a straightforward way (and is something we should be able to just add in an appendix). We were surprised how hard this was for a fair number of the domain-specific models with great log-prob benchmark results on the leaderboard -- ultimately a huge gap between numbers saying 'this is a great medical AI model!' and the ability to use it in production -- and to us that was an important part of the story.

For this set of models where a ton of engineering was required to get workable responses, sharing code is the best we can do. I worry a little about rabbit holing on details of how we could improve tuning or output parsing, because if a model requires so much bespoke effort to work on a task it's been built to perform (in the log-prob terms), the point still stands that you couldn't be confident using it across different types of tasks.

Stepping back, for us this method supported our experience that benchmark performance is pretty disconnected to how a model did with records. This behavior was a big piece of that puzzle that we wanted to show. I think there's some nuance though in how we get this across without getting tied up in the details and options for benchmark hacking.

To your question about the difference between our results and log-prob with T=0: behaviorally, I think of a model like Grok that is tuned to be funny, and perhaps it heavily downweights 'yes' or 'no' on a task like this in favor of saying something entertaining; it may have excellent log-probability benchmark performance, but it would be a much worse choice to power your app than the benchmark scores suggest. We wanted our accuracy to be more reflective of that reality-in-production.

And to your comment about using the phrase state-of-the-art: for us, we _didn't_ want to say "you can get the best model for PubMedQA by doing xyz like we did"; instead, we wanted to say "even if you fully invest in getting great benchmark performance, it doesn't do much for your ability to work with records." So for us, s-o-a is more shorthand for saying "we appropriately exhausted what one can do to tune benchmark performance, and here's a top line number that shows that, so we can stand by the relationship we see between benchmarks and performance on records."

Finally, a last note on something I was seeing yesterday when pawing through some structuring and abstraction tasks that GPT-4o got wrong but LLMD did well. It really is amazing how many different pockets of necessary domain bias/contextual bias the records are teaching the model. One obvious example I was seeing was GPT-4o is undertrained to interpret whether "lab" means "lab test" or "laboratory facility." LLMD has picked up on the association that a task asking for a reference range is referring to a lab test, and that behavior is coming from pre-training and instruction fine-tuning (I suspect more the latter). In contrast, if we don't tune the prompt to be explicit, GPT-4o will start dropping street names into the lab-name outputs, etc.

To me, the implication is that you could do a whack-a-mole approach to load the prompt with ultra precise instructions and it would improve performance on records. But based on what we saw in the paper, that likely _only_ works on the big models like GPT-4o and Sonnet, and not on the domain models that are so hard to coerce into giving reasonable responses. But also, there's a long-tail of such things that would drown you, and so you really have no choice to train on records data. Another tiny example we saw a few weeks ago that has a huge impact on app level performance was that the unit for MCV test is so often wrong in records, but the answer can be assumed to be fL in most cases. So we'd need to add tons of things like that if we didn't have records to train on.

tldr; you need to train on records; if you can't and you have a very well defined purpose/input space, use a big model like GPT-4o and load on the prompt to be very precise -- that should work well; pursuing benchmark performance doesn't get you much practically; if you need to work in an unconstrained environment, you have to train on records to pick up all those small biases that matter.

Thoughts??

We train on real records, and even though they are de-identified in training we still have to keep the model closed and under careful management to protect against the possibility of information leaking.

We are, though, definitely invested in this corner of research, and want to be able to work with others to push medical AI forward.

Given that, the best model for us is to collaborate on an engagement-by-engagement basis. For now we'd look to find ways to do the work directly involving LLMD within our systems.

If you research in the field and have some ideas, I'd love to chat!