Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Defeating Nondeterminism in LLM Inference

thinkingmachines.ai

Post reply on HN
Print view View original

91–100 of 137 posts

Re: Defeating Nondeterminism in LLM Inference

#91

Post by Helmut10001 »

Zacharias030 wrote: post #84
kazinator wrote: post #38

Earlier quoted context omitted.

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

While I get that this is how LLMs work, I think you should think backwards from the user / from what AI as a field is aiming for and recognize that the „naive“ way of the parent to ask for reliable responses no matter what the „context“ is, is exactly what a good AI system should offer. „The context is the input“ betrays a misunderstanding of what (artificial) intelligence systems are aiming for.

Then we need something else. This is not how LLMs work. They are simple statistical predictors, now universal anwsering machines.

Re: Defeating Nondeterminism in LLM Inference

#92

Post by gajjanag »

As others have pointed out, these phenomena are well known to many folks across companies in the AI infra space. It doesn't really break new ground. This article is a good exposition of the basic strategies though.

What I would have loved is a discussion around collectives/multi-node setups. And showing how to get determinism at low performance penalty for multi-node reduction collectives.

Re: Defeating Nondeterminism in LLM Inference

#93

Post by Zacharias030 »

Helmut10001 wrote: post #91
Zacharias030 wrote: post #84

Earlier quoted context omitted.

While I get that this is how LLMs work, I think you should think backwards from the user / from what AI as a field is aiming for and recognize that the „naive“ way of the parent to ask for reliable responses no matter what the „context“ is, is exactly what a good AI system should offer. „The context is the input“ betrays a misunderstanding of what (artificial) intelligence systems are aiming for.

Then we need something else. This is not how LLMs work. They are simple statistical predictors, now universal anwsering machines.

I agree mostly. They are all that you say, but if you think about the conditional distribution that you are learning, there is nothing preventing us in principle from mapping different contexts to the same responses. It is rather a practical limitation that we don’t have sufficient tools of shaping these distributions very soundly. All we can do is throw data at them and hope that they generalize to similar contexts.

We have observed situations where agentic LLM traces on verifiable problems with deterministic (greedy) decoding lead to either completely correct or completely wrong solutions depending on the minutes on the clock which are printed as coincidental output of some tool that the LLM used.

I think there may be some mild fixes to current models available , for example it is worrying that the attention mechanism can never fully disregard any token in the input, because the softmax will always assign a > 0 weight everywhere (and the NN has no way of setting a logit to -infinity). This directly causes that it is extremely difficult for the LLM to fully ignore any part of the context reliably.

However Yann LeCun actually offers some persuasive arguments that autoregressive decoding has some limitations and we may need something better.

Re: Defeating Nondeterminism in LLM Inference

#94

Post by edflsafoiewq »

raincole wrote: post #77
lsy wrote: post #8

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a looku…

> where the exact same input gives different results given different preceding context Why and how is this a problem? If 'preceding context' doesn't cause different results, it means you can simply discard the context. Why do I want that? It's not how I expect a tool to work (I expect vim responds differently to my input after I switch to the insert mode). It's absolutely not how I expect intelligence to work either.…

An example is translation. I MTLed some text recently where the name of a (fictional) city was translated about a dozen different ways. Sometimes you'd get a calque, sometimes you'd get a transliteration (including several wrong ones). Ironically "dumb" MTLs are often much more consistent about this than LLMs.

Re: Defeating Nondeterminism in LLM Inference

#95

Post by orbital-decay »

By setting the temperature to 0 you get greedy decoding, which does a lot more than just making it predictable, and can degrade outputs. Random sampling exists for a reason! Gemini 2.5 Pro in particular doesn't like temp 0, for example.

Focus on correctness, not determinism.

Re: Defeating Nondeterminism in LLM Inference

#96

Post by Zacharias030 »

daralthus wrote: post #50

I thought this was pretty well known (at least in the JAX/XLA world). I've hit this many times and got batch variance explained to me before: https://github.com/google-deepmind/penzai/issues/82 and https://github.com/jax-ml/jax/issues/20047#issuecomment-1975...

should be the top comment.

Re: Defeating Nondeterminism in LLM Inference

#97

Post by matusp »

orbital-decay wrote: post #95

By setting the temperature to 0 you get greedy decoding, which does a lot more than just making it predictable, and can degrade outputs. Random sampling exists for a reason! Gemini 2.5 Pro in particular doesn't like temp 0, for example. Focus on correctness, not determinism.

Determinism does not require temperature=0. You can have a deterministic behavior even with >0 temperature as long as you fix your random seeds.

Re: Defeating Nondeterminism in LLM Inference

#98

Post by dns_snek »

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent.

What LLMs need is the ability to guarantee semantically-equivalent outputs for all semantically-equivalent inputs, but that's very different from "determinism" as we understand it from other algorithms.

Re: Defeating Nondeterminism in LLM Inference

#99

Post by xnx »

mg wrote: post #14

I really hope we will get deterministic LLMs in the future. Even if it causes slightly slower response times. Nondeterminism is what currently keeps me from working with other developers. As I wrote in "Prompt Coding" [1], these days I am not looking for good code. I am looking for prompts that create good code. But how do you share prompts among developers when they produce different code every time? You cannot simp…

> most image models will not create the same image when given the same prompt and parameters.

Really? If you include the seed as one of the parameters most produce pixel identical output.

E.g. "Generate deterministic images" https://cloud.google.com/vertex-ai/generative-ai/docs/image/...

Post reply on HN