>> But the author just took pictures of food & expected a realistic response? Is this genuinely what amounts to a study in AI?
The aim of the study was to understand the variation in results returned by models and how that could cause risks for patients using those models. The main result was measuring within-model variation.
From the pre-print (https://www.diabettech.com/wp-content/uploads/2026/04/diabet...):
We aimed to characterise the within-image reproducibility of carbohydrate estimates from four large language model (LLM) vision APIs and to quantify the clinical risk for insulin dosing, stratifying accuracy by reference value quality.
Methods
Thirteen food photographs were each submitted 495–561 times to four LLM vision
APIs (GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro, Gemini 3.1 Pro Preview) using
an identical structured prompt adapted from the iAPS automated insulin delivery
system (26,904 total queries, temperature 0.01). The primary outcome was within-
image variation (coefficient of variation [CV], range, distributional normality).
Secondary outcomes included accuracy against reference values for nine images,
stratified by quality tier (packet label, weighed/measured, portioned, or visual
estimate). Clinical risk was translated at an insulin-to-carbohydrate ratio of 1:10.
>> I'd like to see this study done using any kind of actual grounding knowledge, seeing what mistakes AI makes when attempting to query ground truth from picture analysis - there would at least be an interesting result methodology in that
The ground truth was established by the author. There's an appendix in the pre-print (Appendix I) that describes the methodology. Methods are described in page 4 of the pre-print:
Reference values for accuracy analysis
For nine of the thirteen images, the author estimated the carbohydrate content using
methods described in Appendix 1. Reference quality was categorised into four tiers:
Tier 1 (packet label): Carbohydrate values derived from manufacturer nutrition
labelling. Two images (cheese sandwich, soup with bread) used bread with
labelled carbohydrate content of 20 g per slice.
Tier 2 (weighed/measured): Portions directly weighed and cross-referenced
with established composition data. Three images (Bakewell tart, bakery cookie,
breakfast burrito).
Tier 3 (portioned): Portions estimated by the author (not weighed) and
combined with USDA composition data. Three images (roast dinner, chilli con
carne with rice, stuffed pork loin).
Tier 4 (visual estimate): Portions and composition estimated from visual
inspection. One image (churros).
For the four restaurant dishes (pizza capricciosa, eggs benedict, crema catalana,
paella), no reference value was established. These images were used for the primary
reproducibility analysis only.
Carbohydrate values follow the EU convention with dietary fibre excluded.