Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Defeating Nondeterminism in LLM Inference

thinkingmachines.ai

Post reply on HN
Print view View original

101–110 of 137 posts

Re: Defeating Nondeterminism in LLM Inference

#101

Post by Ratelman »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

Was my thinking exactly - but also semantically equivalent is also only relevant when it needs to be factual, not necessarily for ALL outputs (if we're aiming for LLM's to present as "human" - or for interactions with LLMs to be natural conversational...). This excludes the world where LLMs act as agents - where you would of course always like the LLM to be factual and thus deterministic.

Re: Defeating Nondeterminism in LLM Inference

#102

Post by Taek »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

You aren't wrong, but that doesn't mean this level of determinism isn't useful. If you don't even have the level of determinism that the exact same input tokens produce the exact same output tokens, then it's very hard to share reproducible results with peers, which can be useful if you are say, red teaming an LLM to produce a very rare / unreliable output.

Re: Defeating Nondeterminism in LLM Inference

#103

Post by atoav »

Nevermark wrote: post #63
riazrizvi wrote: post #56

Earlier quoted context omitted.

Me: What’s an example of a dice roll? LLM: 1 “Language ambiguity with determinism”? Sure I can juxtapose the terms but if it’s semantically inconsistent, then what we mean by that is not a deterministic, definitive thing. You’re chasing your tail on this ‘goal’.

Ambiguity : The request/prompt leaves a lot of room for interpretation. Many qualitatively different answers may be correct, relative to the prompt. Different or non-deterministic models will return highly variance results. Determinism : If a model is given the exact same request/prompt twice, its two responses will also be identical. Whether or not the consistent response qualifies as correct. The two concepts are v…

Maybe I got this wrong but I thought ambiguity refered to the input. So in a deterministic system I would assume that a input of "Give an example of a dice roll" Will always output the exact same example (unless the model also gets the context of the message history).

Ambiguity is what happens when you change the prompt slightly, e.g. by adding a word: "Give an example of a single dice roll". Now as a human our expectation would be that this is the same question and should thus (in a deterministic system) receive the same answer. But to an LLM it may not be.

Re: Defeating Nondeterminism in LLM Inference

#104

Post by qcnguy »

raincole wrote: post #77
lsy wrote: post #8

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a looku…

> where the exact same input gives different results given different preceding context Why and how is this a problem? If 'preceding context' doesn't cause different results, it means you can simply discard the context. Why do I want that? It's not how I expect a tool to work (I expect vim responds differently to my input after I switch to the insert mode). It's absolutely not how I expect intelligence to work either.…

When the context is auto-generated and may include irrelevant data.

This is a common AI benchmark and has been for years before GPT-2 even existed. LLMs need to not get distracted by irrelevant facts and there are tests that measure this. It's the motivation for attention mechanisms, which are the breakthrough that enabled LLMs to scale up.

Re: Defeating Nondeterminism in LLM Inference

#105

Post by helloplanets »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

Not all LLM based applications are a user facing free form chat.

If you take an LLM that makes 10 tool calls in a row for an evaluation, any reduction in unpredictable drift is welcome. Same applies to running your prompt through DSPy Optimizer. [0] Countless other examples. Basically any situation where you are in control of the prompt, the token level input to the LLM, so there's no fuzziness.

In this case, if you would've eliminated token level fuzziness and can yourself guarantee that you're not introducing it from your own end, you can basically map out a much more reliable tree or graph structure of your system's behavior.

[0]: https://dspy.ai/#2-optimizers-tune-the-prompts-and-weights-o...

Re: Defeating Nondeterminism in LLM Inference

#106

Post by menaerus »

> But why aren’t LLM inference engines deterministic? One common hypothesis is that some combination of floating-point non-associativity and concurrent execution leads to nondeterminism based on which concurrent core finishes first. We will call this the “concurrency + floating point” hypothesis for LLM inference nondeterminism. For example, a recent arXiv preprint writes

I'm honored to see that Mira and co. appreciated my feedback on the very topic I made 7 months ago here :D

> You don't need RNG since the whole transformer is an extremely large floating-point arithmetic unit. A wild guess - how about the source of non-determinism is coming from the fact that, on the HW level, tensor execution order is not guaranteed and therefore (T0 * T1) * T2 can produce slightly different results than T0 * (T1 * T2) due to rounding errors?

https://news.ycombinator.com/item?id=42952605#42960047

Re: Defeating Nondeterminism in LLM Inference

#107

Post by mingtianzhang »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

I agree that we need stochasticity in a probabilistic system, but I also think it would be good to control it. For example, we need the stochasticity introduced at high temperatures since it is inherent to the model, but we don’t need stochasticity in matrix computations, as it is not required for modeling.

Re: Defeating Nondeterminism in LLM Inference

#108

Post by frotaur »

kybernetikos wrote: post #60
giveita wrote: post #51

Earlier quoted context omitted.

What's the Weissman score? Or more seriously :) did it perform well. Sounds like it should. If more and more text is AI slop it should do well. I dont fully understand what you said but I guess higher probability logits are encoded with fewer bits. If your text is the LLM output then you may need a bit or two per token?

I used exponential golomb coding, so the rank 0 logit is encoded with a single bit, ranks 1 and 2 are encoded with three bits, ranks 3-6 are encoded with 5 bits, etc. In terms of performance, I've not done any serious testing, but e.g. the wikipedia article on volcanos compresses to about 20% using GPT2. I've seen other strings compress even further. The big issue is that while encoding is not unreasonable, decoding…

I don't know about golomb coding, but with Arithmetic coding you can do stream decoding(AC), if I remember correctly.

I supervised a student's project whose goal was exactly that : implement compression with LLMs using AC.

Since AC is optimal, if your LLM has an average cross entropy x on some dataset, you can expect that the compression will compress data using x nats per token on average!

Re: Defeating Nondeterminism in LLM Inference

#109

Post by redlock »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

Easier to debug deterministic inference

Re: Defeating Nondeterminism in LLM Inference

#110

Post by Dylan16807 »

kazinator wrote: post #38
lsy wrote: post #8

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a looku…

There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people…

> These kinds of requirements betray a misunderstanding of what these LLMs are.

They do not. Refusing to bend your requirements to a system that can't satisfy them is not evidence of misunderstanding the system.

And if you tack on "with X 9s of reliability" then it is something LLMs can do. And in the real world every system has a reliability factor like that.

Post reply on HN