Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Defeating Nondeterminism in LLM Inference

thinkingmachines.ai

Post reply on HN
Print view View original

111–120 of 137 posts

Re: Defeating Nondeterminism in LLM Inference

#111

Post by brisky »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

It would be very useful for AI platform customers. You could run prompts with 0 temperature and check if the results are the same making sure that AI provider is not switching the PRO model in the background for a cheap one and ripping you off.

Re: Defeating Nondeterminism in LLM Inference

#113

Post by Dylan16807 »

nomel wrote: post #67
bee_rider wrote: post #66

Earlier quoted context omitted.

I mean if you wait long enough, it is asking for .abs().max().item() of something that can be identified as definitionally zero.

I don't understand. Since it's not using the parallel interface, only one operation can happen at a time. This would be, literally, sequential execution with extra overhead, in this case . Again, in this case , what would hope to be achieved from doing things lazily, since the lazy operations would immediately be followed by their evaluation? The parallel interface, which is async, is probably what you're lookin for.

Let's look at the subtraction in this case.

If evaluation is lazy, then the subtraction operator gets fed two unevaluated matrix multiplies.

If it's a dumb subtraction operator, this gives us no benefit. Eventually it evaluates both and then subtracts. And it has some extra overhead like you said.

But if it's a smart subtraction operator, it can realize that both parameters are the same equation, and then it can return all 0s without evaluating anything.

And even better than just skipping the matrix math, "all 0s" can be a stub object that takes O(1) time to set up. And then .abs().max() will be instant too.

Re: Defeating Nondeterminism in LLM Inference

#114

Post by virgilp »

Helmut10001 wrote: post #91
Zacharias030 wrote: post #84

Earlier quoted context omitted.

While I get that this is how LLMs work, I think you should think backwards from the user / from what AI as a field is aiming for and recognize that the „naive“ way of the parent to ask for reliable responses no matter what the „context“ is, is exactly what a good AI system should offer. „The context is the input“ betrays a misunderstanding of what (artificial) intelligence systems are aiming for.

Then we need something else. This is not how LLMs work. They are simple statistical predictors, now universal anwsering machines.

> They are simple statistical predictors, now universal anwsering machines.

I see this a lot. I kinda' doubt the "simple" part, but even beyond that, is there any evidence that statistical predictor can't be a universal answering machine? I think there's plenty of evidence that our thinking is at least partially a statistical predictor (e.g. when you see a black sheep you don't think "at least one side of this sheep is black", you fully expect it to be black on both sides)

I'm not saying that LLMs _are_ universal answering machines. I'm wondering why people question that they are/they can become one, based on the argument that "fundamentally they are statistical predictors". So they are. So what?

Re: Defeating Nondeterminism in LLM Inference

#115

Post by gtsop »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

When you do MCP-style applications, an LLM is more like RegEx on steroids, and since you expect your regex to return the same matches on the same input, it is a very desirable attribute for LLMs as well. I would say it is more than desirable, it is necessary.

If i want to covert "how do I x" to `api.howTo("x")` it is very important that i get the exact same result every time.

Re: Defeating Nondeterminism in LLM Inference

#116

Post by ZeljkoS »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

For "bug" reproduction purposes. It is easier to debug a model if the same string always produces the same incorrect or strange LLM output, not every 100th time you run it.

Re: Defeating Nondeterminism in LLM Inference

#117

Post by ontouchstart »

ZeljkoS wrote: post #116
dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

For "bug" reproduction purposes. It is easier to debug a model if the same string always produces the same incorrect or strange LLM output, not every 100th time you run it.

If there is a bug (a behavior defined by whatever criteria), it is just a single path in a very complex systems with high connectivity.

This nonlinear and chaotic behavior regardless of implementation details of the black box makes LLM seem to be nondeterministic. But LLM is just a pseudo random number generator with a probability distribution.

(As I am writing this on my iPhone with text completion, I can see this nondeterministic behavior)

Re: Defeating Nondeterminism in LLM Inference

#118

Post by kodablah »

dns_snek wrote: post #98

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…

Deterministic output is needed when LLMs are used for validations. This can be anything from input validation at runtime to a CI check leveraging LLMs. It can be argued this is not an acceptable use of AI, but it will become increasingly common and it will need to be tweaked/tested. You cannot tweak/test a response you don't know you're going to get.

Re: Defeating Nondeterminism in LLM Inference

#119

Post by sailingparrot »

sudohalt wrote: post #87
sailingparrot wrote: post #24

Earlier quoted context omitted.

I suggest you look up the name of the main author of TFA before assuming they don’t know what they are talking about. This is literally one of the most knowledgeable person on the topic. I think you are the one that hasn’t peeled enough layers to connect with what they are saying.

1. they aren't, they are just popular online. 2. the author has nothing to do with the original comment. Why do you think academic reviews are double blind?

One of the top 5 most active contributors to pytorch over the last few years, and specifically working on some of it's most hardcore components is "just popular online"?

If you say so.

> the author has nothing to do with the original comment

Except for the part of the comment that was assuming the author had no idea how this all works, has only used LLMs through API and has never run a local model, you mean?

Re: Defeating Nondeterminism in LLM Inference

#120

Post by sailingparrot »

golol wrote: post #32
oasisaimlessly wrote: post #25

Earlier quoted context omitted.

It's possible to deterministically sample from a probability distribution. For example, just seed your RNG with a constant, or with the SHA256 hash of the context.

Well yes, you can "hack" the pseudorandom number generator, but... that's not really the point when talking about determinism in LLMs is it? I mean the mathematical idea of the standard LLM is certainly truly random.

> I mean the mathematical idea of the standard LLM is certainly truly random.

Not really, LLMs give you a distribution over possible next tokens. You are free to then sample from this distribution how you want. There is no need to hack RNG or whatever, for example you can simply just take a greedy approach and always output the most likely token, in which case the LLM becomes deterministic (mathematically). This is equivalent to setting the temperature to 0.

Post reply on HN