Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Defeating Nondeterminism in LLM Inference

thinkingmachines.ai

Post reply on HN
Print view View original

31–40 of 137 posts

Re: Defeating Nondeterminism in LLM Inference

#31

Post by khimaros »

mg wrote: post #14

I really hope we will get deterministic LLMs in the future. Even if it causes slightly slower response times. Nondeterminism is what currently keeps me from working with other developers. As I wrote in "Prompt Coding" [1], these days I am not looking for good code. I am looking for prompts that create good code. But how do you share prompts among developers when they produce different code every time? You cannot simp…

i tried to create a makefile driven workflow based on this idea and ended up with https://github.com/khimaros/enc -- it suffers from the issues you raised

i'm hoping that it becomes more useful as models improve and become more reliable at producing working code (though determinism would be great for improving prompts).

Re: Defeating Nondeterminism in LLM Inference

#32

Post by golol »

oasisaimlessly wrote: post #25
golol wrote: post #23

Earlier quoted context omitted.

Hold on a second. A transformer produces deterministically a probability distribution over the token alphabet from the context. Then one samples from this distribution. This is random and meant to be random.

It's possible to deterministically sample from a probability distribution. For example, just seed your RNG with a constant, or with the SHA256 hash of the context.

Well yes, you can "hack" the pseudorandom number generator, but... that's not really the point when talking about determinism in LLMs is it? I mean the mathematical idea of the standard LLM is certainly truly random.

Re: Defeating Nondeterminism in LLM Inference

#33

Post by eldenring »

boroboro4 wrote: post #27
eldenring wrote: post #18

Very impressive! I guess this still wouldn't affect their original example > For example, you might observe that asking ChatGPT the same question multiple times provides different results. even with 0.0 temperature due to MOE models routing at a batch level, and you're very unlikely to get a deterministic batch. > Not because we’re somehow leaking information across batches — instead, it’s because our forward pass la…

> even with 0.0 temperature due to MOE models routing at a batch level, and you're very unlikely to get a deterministic batch. I don’t think this is correct - MoE routing happens at per token basis. It can be non deterministic and batch related if you try to balance out your experts load in a batch but that’s performance optimization (just like all of the blogpost) and not the way models are trained to work.

Ah interesting, good point. So I guess expert-choice routing leaks across the batch. Now I'm not sure.

Re: Defeating Nondeterminism in LLM Inference

#34

Post by threeducks »

It should also be noted that PyTorch has a page about reproducibility: https://docs.pytorch.org/docs/stable/notes/randomness.html

TL;DR

Seed your PRNGs and call torch.use_deterministic_algorithms(True) to get the deterministic kernels. They may be slightly slower, but in practice, you probably will not notice.

Note that results will still differ between different drivers and GPUs. It would be great if NVIDIA tried harder in that regard.

Re: Defeating Nondeterminism in LLM Inference

#35

Post by nowittyusername »

golol wrote: post #23
nowittyusername wrote: post #21

I am baffled that I still run against these statement years after LLM's have been around. LLM's are deterministic and always have been. The reason people are having issues with them is because they are basing their assumptions on api based experiments. Like my man, how can you be making these statements when you haven't done the due diligence of running the LLM on your own hardware with all of the variables locked do…

Hold on a second. A transformer produces deterministically a probability distribution over the token alphabet from the context. Then one samples from this distribution. This is random and meant to be random.

The sampling process isn't random. If you sample with identical sampling parameters and identical values for said parameters, you will always get same results. You only start getting "non deterministic" behavior when you start using more complex systems outside the scope of your control like multi gpu systems and batch processing. One llm sampled with cash prompting off and and batch processing off will always generate same results if all values are same.

Re: Defeating Nondeterminism in LLM Inference

#38

Post by kazinator »

lsy wrote: post #8

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a looku…

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input!

If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context.

Now what some people want is requirements like:

- The different wording of a prompt with exactly the same meaning should not change anything in the output; e.g. whether you say "What is the capital of France" or "What is France's capital" the answer should be verbatim identical.

- Prior context should not change responses in ways that don't have any interaction with the context. For instance, a prompt is given "what is 2 + 2", then the answer should always be the same, except if the context instructs the LLM that 2 + 2 is to be five.

These kinds of requirements betray a misunderstanding of what these LLMs are.

Re: Defeating Nondeterminism in LLM Inference

#39

Post by p1necone »

mg wrote: post #14

I really hope we will get deterministic LLMs in the future. Even if it causes slightly slower response times. Nondeterminism is what currently keeps me from working with other developers. As I wrote in "Prompt Coding" [1], these days I am not looking for good code. I am looking for prompts that create good code. But how do you share prompts among developers when they produce different code every time? You cannot simp…

Surely if you end up relying on a given prompt to produce the exact same code every time you should instead just check that code into source control the first time you generate it?

A deterministic LLM isn't going to behave appreciably differently from a non deterministic one if your input or context varies by even a tiny bit (pun intended) each time.

Post reply on HN