Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Defeating Nondeterminism in LLM Inference

thinkingmachines.ai

Post reply on HN
Print view View original

61–70 of 137 posts

Re: Defeating Nondeterminism in LLM Inference

#61

Post by nakamoto_damacy »

"in collaboration with others at Thinking Machines"

If you're old enough, you might remember Danny Hillis' Thinking Machines from the late 80s. I wish they had chosen a different name (I say this for nostalgic reasons, having been in front of one of those cubes glowing with red LEDs back in the late 80s at MIT's AI Lab" (renamed to CSAIL at some point). Feynman did some amazing work on that, too: https://longnow.org/ideas/richard-feynman-and-the-connection...

In the U.S., the “THINKING MACHINES” trademarks were owned by Thinking Machines Corporation (the company Hillis co-founded), not Hillis personally, and those registrations were cancelled in 1998–1999. USPTO Report +1

The company itself went bankrupt in 1994 and its assets were dispersed (e.g., to Sun Microsystems, later Oracle).

There’s a new, pending USPTO application for “THINKING MACHINES” filed in 2025 by Thinking Machines Lab Inc., the company founded by Amira Murati.

Re: Defeating Nondeterminism in LLM Inference

#62

Post by skybrian »

kazinator wrote: post #38
lsy wrote: post #8

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a looku…

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

I wonder if there's a way to use an LLM to rewrite the prompt, standardizing the wording when two prompts mean the same thing?

Re: Defeating Nondeterminism in LLM Inference

#63

Post by Nevermark »

riazrizvi wrote: post #56
atoav wrote: post #53

Earlier quoted context omitted.

Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow. If I ask the same model the same question I should be able to deterministically get the same answer. Now if we phrase the same question slightly differently we would expect to get a slightly different answer.

Me: What’s an example of a dice roll? LLM: 1 “Language ambiguity with determinism”? Sure I can juxtapose the terms but if it’s semantically inconsistent, then what we mean by that is not a deterministic, definitive thing. You’re chasing your tail on this ‘goal’.

Ambiguity: The request/prompt leaves a lot of room for interpretation. Many qualitatively different answers may be correct, relative to the prompt. Different or non-deterministic models will return highly variance results.

Determinism: If a model is given the exact same request/prompt twice, its two responses will also be identical. Whether or not the consistent response qualifies as correct.

The two concepts are very different.

(Ambiguous vs. precise prompt) x (Deterministic vs. Non-deterministic model) = 4 different scenarios.

A model itself can be non-deterministic without being ambiguous. If you know exactly how it functions, why it is non-deterministic (batch sensitive for instance), that is not an ambiguous model. Its operation is completely characterized. But it is non-deterministic.

An ambiguous model would simply be model whose operation was not characterized. A black box model for instance. A black box model can be deterministic and yet ambiguous.

Re: Defeating Nondeterminism in LLM Inference

#64

Post by d4mi3n »

skybrian wrote: post #62
kazinator wrote: post #38

Earlier quoted context omitted.

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

I wonder if there's a way to use an LLM to rewrite the prompt, standardizing the wording when two prompts mean the same thing?

Not an expert, but I've been told RAG in combination with a database of facts is one way to get more consistency here. Using one of the previous examples, you might have a knowledge store (usually a vector database of some kind) that contains a mapping of countries to capitols and the LLM would query it whenever it had to come up with an answer rather than relying on whatever was baked into the base model.

Re: Defeating Nondeterminism in LLM Inference

#65

Post by ewoodrich »

ricardobeat wrote: post #52
gond wrote: post #48

I am still irritated by the name of the company. What is the reasoning behind these schemes? The hope that bits of the properties of legendary companies will rub off onto the new venture? As if naming the next best venture PARC will inevitably create a breakthrough in networking just by the arrangement of four letters.

Are you talking about the “Thinking Machines” company that shut down in 1994? Took me some digging to figure it out, doesn’t seem well-known enough to be the reason - it’s just a nice (and relatively obvious) name.

It may not be a household name like Apple or Microsoft but its flagship product the Connection Machine is somewhat iconic in (super)computing history. The physical design of the machine is cool and unforgettable looking, plus recurring HN favorite Richard Feynman contributed to the original architecture.

Re: Defeating Nondeterminism in LLM Inference

#66

Post by bee_rider »

nomel wrote: post #47
bee_rider wrote: post #46

From their code: A = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16) B = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16) ref = torch.mm(A, B) for _ in range(1000): assert (torch.mm(A, B) - ref).abs().max().item() == 0 I’m sort of surprised that Torch doesn’t have some kind of lazy evaluation thing to avoid computing anything here. I thought that was one of the nice things about all these fan…

Maybe I'm missing something, but in this case, wouldn't being lazy would be pure overhead? I don't see anything can be lazy here. The reference computed once, nanoseconds before it's needed, and test cases computed at the time of comparison, then tossed away. What would hope to be achieved by making this case lazy? If you wanted these to run in parallel, with a multi-gpu system, you would use the appropriate parallel…

I mean if you wait long enough, it is asking for

  .abs().max().item()
of something that can be identified as definitionally zero.

Re: Defeating Nondeterminism in LLM Inference

#67

Post by nomel »

bee_rider wrote: post #66
nomel wrote: post #47

Earlier quoted context omitted.

Maybe I'm missing something, but in this case, wouldn't being lazy would be pure overhead? I don't see anything can be lazy here. The reference computed once, nanoseconds before it's needed, and test cases computed at the time of comparison, then tossed away. What would hope to be achieved by making this case lazy? If you wanted these to run in parallel, with a multi-gpu system, you would use the appropriate parallel…

I mean if you wait long enough, it is asking for .abs().max().item() of something that can be identified as definitionally zero.

I don't understand. Since it's not using the parallel interface, only one operation can happen at a time. This would be, literally, sequential execution with extra overhead, in this case. Again, in this case, what would hope to be achieved from doing things lazily, since the lazy operations would immediately be followed by their evaluation?

The parallel interface, which is async, is probably what you're lookin for.

Re: Defeating Nondeterminism in LLM Inference

#68

Post by Jensson »

atoav wrote: post #53
riazrizvi wrote: post #41

Natural language is ambiguous. It needs to be. I think the approach here of trying to figure out how to make circles into squares, and argue why circles should be squares, is misguided. Discussions of this type are going to eventually morph into better understanding of how to accept ambiguity and randomness in language, and further shape it with other larger sub-patterns beyond the little proto-grammars that the QKV…

Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow. If I ask the same model the same question I should be able to deterministically get the same answer. Now if we phrase the same question slightly differently we would expect to get a slightly different answer.

> Now if we phrase the same question slightly differently we would expect to get a slightly different answer.

You wouldn't get this from an LLM though, a tiny change in starting point gets a massive change in output, its a chaotic system.

Re: Defeating Nondeterminism in LLM Inference

#69

Post by kazinator »

skybrian wrote: post #62
kazinator wrote: post #38

Earlier quoted context omitted.

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

I wonder if there's a way to use an LLM to rewrite the prompt, standardizing the wording when two prompts mean the same thing?

It's going to backfire. In real scenarios (not regression testing) users don't want to see the exact same thing twice out of the LLM in the same session in spite of trying to refine the result with more context.

There are going to be false positives: text that is subtly different from a previous response is misidentified as a duplicate such that the previous response is substituted for it, frustrating the user.

Re: Defeating Nondeterminism in LLM Inference

#70

Post by skybrian »

riazrizvi wrote: post #56
atoav wrote: post #53

Earlier quoted context omitted.

Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow. If I ask the same model the same question I should be able to deterministically get the same answer. Now if we phrase the same question slightly differently we would expect to get a slightly different answer.

Me: What’s an example of a dice roll? LLM: 1 “Language ambiguity with determinism”? Sure I can juxtapose the terms but if it’s semantically inconsistent, then what we mean by that is not a deterministic, definitive thing. You’re chasing your tail on this ‘goal’.

If you really want that to work while being reproducible, maybe give it a random number tool and set the seed?
Post reply on HN