Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Defeating Nondeterminism in LLM Inference

thinkingmachines.ai

Post reply on HN
Print view View original

71–80 of 137 posts

Re: Defeating Nondeterminism in LLM Inference

#71

Post by skybrian »

p1necone wrote: post #39
mg wrote: post #14

I really hope we will get deterministic LLMs in the future. Even if it causes slightly slower response times. Nondeterminism is what currently keeps me from working with other developers. As I wrote in "Prompt Coding" [1], these days I am not looking for good code. I am looking for prompts that create good code. But how do you share prompts among developers when they produce different code every time? You cannot simp…

Surely if you end up relying on a given prompt to produce the exact same code every time you should instead just check that code into source control the first time you generate it? A deterministic LLM isn't going to behave appreciably differently from a non deterministic one if your input or context varies by even a tiny bit (pun intended) each time.

If nothing has changed, caching the result would certainly be cheaper. But if you're doing that as part of a test, it's not really running the test and it might defeat the purpose of the test.

Re: Defeating Nondeterminism in LLM Inference

#72

Post by taneq »

skybrian wrote: post #62
kazinator wrote: post #38

Earlier quoted context omitted.

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

I wonder if there's a way to use an LLM to rewrite the prompt, standardizing the wording when two prompts mean the same thing?

Deterministically, you mean? ;)

Re: Defeating Nondeterminism in LLM Inference

#73

Post by skybrian »

kazinator wrote: post #69
skybrian wrote: post #62

Earlier quoted context omitted.

I wonder if there's a way to use an LLM to rewrite the prompt, standardizing the wording when two prompts mean the same thing?

It's going to backfire. In real scenarios (not regression testing) users don't want to see the exact same thing twice out of the LLM in the same session in spite of trying to refine the result with more context. There are going to be false positives: text that is subtly different from a previous response is misidentified as a duplicate such that the previous response is substituted for it, frustrating the user.

Google search rewrites misspelled search queries and also lets you override it if that's not what you want. Maybe something similar would work?

Re: Defeating Nondeterminism in LLM Inference

#74

Post by bee_rider »

nomel wrote: post #67
bee_rider wrote: post #66

Earlier quoted context omitted.

I mean if you wait long enough, it is asking for .abs().max().item() of something that can be identified as definitionally zero.

I don't understand. Since it's not using the parallel interface, only one operation can happen at a time. This would be, literally, sequential execution with extra overhead, in this case . Again, in this case , what would hope to be achieved from doing things lazily, since the lazy operations would immediately be followed by their evaluation? The parallel interface, which is async, is probably what you're lookin for.

The Python commands are encountered sequentially. One could image a library where the Python commands build the computation under the hood. Then, the library would be able to take advantage of situations like this one (or, more practically, reorder multiplications and/or avoid unnecessary temporaries).

Re: Defeating Nondeterminism in LLM Inference

#75

Post by Imnimo »

nakamoto_damacy wrote: post #61

"in collaboration with others at Thinking Machines" If you're old enough, you might remember Danny Hillis' Thinking Machines from the late 80s. I wish they had chosen a different name (I say this for nostalgic reasons, having been in front of one of those cubes glowing with red LEDs back in the late 80s at MIT's AI Lab" (renamed to CSAIL at some point). Feynman did some amazing work on that, too: https://longnow.org/…

I make this mistake every time I see their name.

Re: Defeating Nondeterminism in LLM Inference

#76

Post by lrvick »

saagarjha wrote: post #9
lrvick wrote: post #7

Job one is have every bit of software involved also be deterministic, which stagex takes care of. I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago. Run two of these with the same prompts and same seed and you get the same results. Obviously in GPU clusters with different hardware things get more complicated. https://git.distrust.co/public/llmshell

What’s stagex?

supply chain security focused linux distro that does not trust its own maintainers by design.

Re: Defeating Nondeterminism in LLM Inference

#77

Post by raincole »

lsy wrote: post #8

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a looku…

> where the exact same input gives different results given different preceding context

Why and how is this a problem?

If 'preceding context' doesn't cause different results, it means you can simply discard the context. Why do I want that? It's not how I expect a tool to work (I expect vim responds differently to my input after I switch to the insert mode). It's absolutely not how I expect intelligence to work either. It sounds like the most extreme form of confirmation bias.

Re: Defeating Nondeterminism in LLM Inference

#78

Post by raincole »

riazrizvi wrote: post #56
atoav wrote: post #53

Earlier quoted context omitted.

Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow. If I ask the same model the same question I should be able to deterministically get the same answer. Now if we phrase the same question slightly differently we would expect to get a slightly different answer.

Me: What’s an example of a dice roll? LLM: 1 “Language ambiguity with determinism”? Sure I can juxtapose the terms but if it’s semantically inconsistent, then what we mean by that is not a deterministic, definitive thing. You’re chasing your tail on this ‘goal’.

> LLM: 1

A perfectly acceptable answer.

If it answers 1 every time it's still a perfectly acceptable answer.

Re: Defeating Nondeterminism in LLM Inference

#79

Post by kjkjadksj »

kazinator wrote: post #38
lsy wrote: post #8

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a looku…

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

If an input along with the context generated some random seed or hash this would certainly be possible. Just paste your seed over to your coworker, they supply it to the model and it contains all contextual information.
Post reply on HN