He asked AI to count carbs 27000 times. It couldn't give the same answer twice

Post by azakai » Wed, Apr 29, 2026, 3:55 PM UTC

notahacker wrote: post #179

endymion-light wrote: post #159

Earlier quoted context omitted.

But I don't see them using those commercial services in this study - instead, they're using frontier model companies? Is Gemini advertising that you get a realistic calorie count from a picture? Maybe so - in which case i'd take it back!

The commercial services likely also have frontier model dependencies... The opening to the actual paper is quite explicit that (i) other studies have already tested commercial apps with with unimpressive results and (ii) a popular open source app for carb counting directly relies on API calls from these frontier models, and this research batch tested the images used the exact same models and prompts as the popular op…

Post by muwtyhg » Wed, Apr 29, 2026, 4:00 PM UTC

endymion-light wrote: post #210

datsci_est_2015 wrote: post #197

Earlier quoted context omitted.

The obvious meme to invoke here is: - AI will solve all of our problems - No not like that! Are the trillion dollars sloshing around the AI economy well-invested if the refrain is always “you’re holding it wrong”? So we’re trying to define, through trial and error, what problems “AI” will actually solve, and this paper is one of the many cobblestones on that road.

i mean it's more like "AI can solve this one problem, but it needs X, Y, Z, because it's not a omnipotent god entity" "I tried it without any of those things and it didn't work - this is worthless tech!" I don't know if more accurate calorie counting using AI exists - but it's like being upset that the screwdriver isn't gluing wood. AI is far more than frontier LLMs.

Post by muwtyhg » Wed, Apr 29, 2026, 4:12 PM UTC

pkaye wrote: post #258

bcjdjsndon wrote: post #202

Earlier quoted context omitted.

But why would the same llm give you wildly different answers EACH TIME you ask?

There is a parameter in LLMs called temperature that controls creativity/randomness. If you set it to 0 it makes the model deterministic. I think some LLMs expose this as a tunable parameter.

Post by muwtyhg » Wed, Apr 29, 2026, 4:15 PM UTC

falcor84 wrote: post #81

Ekaros wrote: post #36

Earlier quoted context omitted.

Then it should refuse to answer 100% of time.

I don't think refusal is the right approach. I would much prefer that it respond with something like: > There is not enough information to make an accurate estimate, but if you'd like, I can take a stab at it. If so, how much effort to put into it? > Yes, go ahead and spend up to 5mins and $1 to analyze it. > Done, I've had 100 subagents analyze the image and have arrived at a 95% confidence interval of the portion c…

Post by pertymcpert » Wed, Apr 29, 2026, 4:18 PM UTC

endymion-light wrote: post #218

chromacity wrote: post #208

Earlier quoted context omitted.

> There's an incredibly serious lack of education with how LLMs & carb-counting works Oh! Do the vendors offer trainings to make sure the users understand how LLMs work? If not, surely, the LLM itself is trained to know its limitations and politely decline in situations like that?... The #1 use case for this tech is "here's a problem I don't feel like solving, let's have a computer do magic ". It's how it's advertise…

yes - the vast majority of labs offer a whole host of training material for users to understand how LLMs work. There's entire course websites created by each of the major vendors specifically to understand how LLMs work. Here's a couple of examples: https://academy.openai.com/public/content https://www.commonsense.org/education/articles/practical-tip... Quote >Getting the most out of generative AI depends on what you…