Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A new benchmark for testing LLMs for deterministic outputs

interfaze.ai

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Show HN: A new benchmark for testing LLMs for deterministic outputs

#31

Post by nemo1618 »

iLoveOncall wrote: post #23
nemo1618 wrote: post #19

Earlier quoted context omitted.

LLMs are not inherently non-deterministic. This is a common misconception. You used to be able to set temp=0 and a fixed seed and get the same output every time. This broke when labs started implementing batching, and no one bothered fixing it because the benefits of batching vastly outweighed the demand for deterministic output. I am hopeful deterministic output will return, though; DeepSeek v4 claims to have implem…

> LLMs are not inherently non-deterministic. Reproducible does not mean deterministic. You cannot determine in advance what a prompt will give as output, even with a temperature of 0 and a fixed seed, therefore they are not deterministic.

Huh? I'm not aware of anyone else who defines "deterministic" that way. "Deterministic" comes from "determinism," as in "the effects are fully determined by the causes" -- not "determine" as in "deduce."

Re: Show HN: A new benchmark for testing LLMs for deterministic outputs

#34

Post by khurdula »

Flux159 wrote: post #3
stared wrote: post #2

Thank you for sharing benchmark. However, the results are selective. Why no Opus 4.7? Why Gemini 3.1 Pro is missing? If there is some other criterion (e.g. models within certain time or budget), great - just make it explicit. When I see "Top 5 at a glance" and it missed key frontier models, I am (at best) confused.

Agree that the choices are strange. Sonnet 4.6 was tested, but no Opus 4.6. Gemini 3.1 and GLM 5 came out around the same time as Sonnet 4.6 (~Feb 2026) so it's strange that they are missing, but Gemini 2.5 Flash, Gemini 3 Flash, and GLM 4.7 are there.

We've updated our leaderboard having evaluated frontier models gemini 3.1 pro, opus 4.6 & 4.7, glm 5.1, deepseek v4, Kimi K2.6 as well.

Re: Show HN: A new benchmark for testing LLMs for deterministic outputs

#35

Post by khurdula »

maxdo wrote: post #15

gpt 5.5 seems to be the recent leader overall, it make sense to include it , just to see what you trade off for speed/open source nature vs cutting edge leader.

hey! we've evaluated gpt 5.5 as well along with other frontier models. gemini and gemma models outperform it across all three modalities.

Open source models like glm 4.7 still compete closely with table toppers.

Re: Show HN: A new benchmark for testing LLMs for deterministic outputs

#37

Post by celloer »

Cool work — quick question: how should readers think about the fact that Interfaze-Beta is on the leaderboard you built? Not saying anything's wrong with the methodology, just curious how you'd recommend a third party verify the ranking is neutral to the choices you made (datasets, difficulty weights, reasoning-off default, etc.).

Re: Show HN: A new benchmark for testing LLMs for deterministic outputs

#38

Post by khurdula »

celloer wrote: post #37

Cool work — quick question: how should readers think about the fact that Interfaze-Beta is on the leaderboard you built? Not saying anything's wrong with the methodology, just curious how you'd recommend a third party verify the ranking is neutral to the choices you made (datasets, difficulty weights, reasoning-off default, etc.).

We've open-sourced all code, and test sets. You can find them here: https://interfaze.ai/blog/introducing-structured-output-benc...

To validate the choices and configurations, feel free to give it a reading. We also breakdown our methodology in the blog and in-depth within the paper.

Post reply on HN