Defeating Nondeterminism in LLM Inference
thinkingmachines.ai
Defeating Nondeterminism in LLM Inference
1–10 of 137 posts
Re: Defeating Nondeterminism in LLM Inference
#2I think this means that the results might also be non-deterministic across hardware revisions b/c I don't think they verified that the kernels will work the same on different GPU & TPU versions b/c how do they know that the compiler will not re-order the operations behind their back?
Re: Defeating Nondeterminism in LLM Inference
#3I think this means that the results might also be non-deterministic across hardware revisions b/c I don't think they verified that the kernels will work the same on different GPU & TPU versions b/c how do they know that the compiler will not re-order the operations behind their back?
Ensuring the same floating-point algorithm workload behaves exactly the same on two distinct workstations is a heck of a lot of work that almost no one is willing to pay for.
Re: Defeating Nondeterminism in LLM Inference
#4I think this means that the results might also be non-deterministic across hardware revisions b/c I don't think they verified that the kernels will work the same on different GPU & TPU versions b/c how do they know that the compiler will not re-order the operations behind their back?
[deleted]
Re: Defeating Nondeterminism in LLM Inference
#5I think this means that the results might also be non-deterministic across hardware revisions b/c I don't think they verified that the kernels will work the same on different GPU & TPU versions b/c how do they know that the compiler will not re-order the operations behind their back?
Ensuring the same floating-point algorithm workload behaves exactly the same on two distinct workstations is a heck of a lot of work that almost no one is willing to pay for.
Not only that but heterogeneous clusters (inevitable at a large enough scale) will also have non-deterministic outputs. So it's great that they wrote kernels to make the forward pass deterministic but getting rid of it entirely at data center scale would mean that they'd also have to do this type of work across cluster nodes as well to maintain "cluster" invariance & not just batch invariance.
Re: Defeating Nondeterminism in LLM Inference
#6I think this means that the results might also be non-deterministic across hardware revisions b/c I don't think they verified that the kernels will work the same on different GPU & TPU versions b/c how do they know that the compiler will not re-order the operations behind their back?
> will not re-order the operations behind their back?
Valid point. Floating point summation is not always commutative.
Re: Defeating Nondeterminism in LLM Inference
#7Job one is have every bit of software involved also be deterministic, which stagex takes care of.
I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago.
Run two of these with the same prompts and same seed and you get the same results.
Obviously in GPU clusters with different hardware things get more complicated.
Re: Defeating Nondeterminism in LLM Inference
#8Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result.
Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a lookup table would do just as well. You can't use "correct" unit tests or evaluation sets to prove anything about inputs you haven't tested.
Re: Defeating Nondeterminism in LLM Inference
#9Job one is have every bit of software involved also be deterministic, which stagex takes care of. I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago. Run two of these with the same prompts and same seed and you get the same results. Obviously in GPU clusters with different hardware things get more complicated. https://git.distrust.co/public/llmshell
What’s stagex?
Re: Defeating Nondeterminism in LLM Inference
#10Job one is have every bit of software involved also be deterministic, which stagex takes care of. I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago. Run two of these with the same prompts and same seed and you get the same results. Obviously in GPU clusters with different hardware things get more complicated. https://git.distrust.co/public/llmshell
That's not what this is about.
"I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago" looks like you're using llama-cpp in that repo. This is about vllm serving many requests at once, at long sequence lengths.
> As it turns out, our request’s output does depend on the parallel user requests. Not because we’re somehow leaking information across batches — instead, it’s because our forward pass lacks “batch invariance”, causing our request’s output to depend on the batch size of our forward pass.
Your situation isn't really comparable.