[deleted]
Defeating Nondeterminism in LLM Inference
121–130 of 137 posts
Re: Defeating Nondeterminism in LLM Inference
#122Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…
I'm actually working on something similar to this where you can encode information into the outputs of LLM's via steganography: https://github.com/sutt/innocuous
Since I'm really looking to sample the only the top ~10 tokens, and I mostly test on CPU-based inference of 8B models, there's probably not a lot of worries getting a different order of the top tokens based on hardware implementation, but I'm still going to take a look at it eventually, and build in guard conditions against any choice that would be changed by an epsilon of precision loss.
Re: Defeating Nondeterminism in LLM Inference
#123This is eternal struggle. - Hardware developers will constantly scale horizontally and make less (time) deterministic hardware, because wall of memory, and scientists could constantly develop new ways to make calculations deterministic.
So, even if will be achieved progress just now, I think in predictable future this will be constant dead-end.
Re: Defeating Nondeterminism in LLM Inference
#124Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…
Deterministic output is needed when LLMs are used for validations. This can be anything from input validation at runtime to a CI check leveraging LLMs. It can be argued this is not an acceptable use of AI, but it will become increasingly common and it will need to be tweaked/tested. You cannot tweak/test a response you don't know you're going to get.
yeah indeed, regression testing for chatbots that use RAGs would involve making sure the correct response comes from the RAG.
Today we have a extremely hacky workaround by ensuring that at least the desired chunk from the RAG is selected, but it's far from ideal and our code is not well written (a temporary POC written by AI that has been there for quite some months now ...)
Re: Defeating Nondeterminism in LLM Inference
#125Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…
Not all LLM based applications are a user facing free form chat. If you take an LLM that makes 10 tool calls in a row for an evaluation, any reduction in unpredictable drift is welcome. Same applies to running your prompt through DSPy Optimizer. [0] Countless other examples. Basically any situation where you are in control of the prompt, the token level input to the LLM, so there's no fuzziness. In this case, if you…
> If you take an LLM that makes 10 tool calls in a row for an evaluation, any reduction in unpredictable drift is welcome
why use an ambiguous natural language for a specific technical task? i get that its a cool trick but surely they can come up with another input method by now?
Re: Defeating Nondeterminism in LLM Inference
#126"in collaboration with others at Thinking Machines" If you're old enough, you might remember Danny Hillis' Thinking Machines from the late 80s. I wish they had chosen a different name (I say this for nostalgic reasons, having been in front of one of those cubes glowing with red LEDs back in the late 80s at MIT's AI Lab" (renamed to CSAIL at some point). Feynman did some amazing work on that, too: https://longnow.org/…
[flagged]
Re: Defeating Nondeterminism in LLM Inference
#127"in collaboration with others at Thinking Machines" If you're old enough, you might remember Danny Hillis' Thinking Machines from the late 80s. I wish they had chosen a different name (I say this for nostalgic reasons, having been in front of one of those cubes glowing with red LEDs back in the late 80s at MIT's AI Lab" (renamed to CSAIL at some point). Feynman did some amazing work on that, too: https://longnow.org/…
[flagged]
I had no idea. But I believe the same deal with Einstein being a dick to his wife and never acknowledging his friend who taught him the math he used in his work (I read about that recently from a respectable source.) I guess that makes sense; no one is void of some deep flaw, it's just selectively hidden.
Re: Defeating Nondeterminism in LLM Inference
#128Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent o…
I don't think the claim is that this is particularly helpful for consumer-facing applications. But from a research perspective, this is invaluable for allowing reproducibility.
Re: Defeating Nondeterminism in LLM Inference
#129Earlier quoted context omitted.
I don't understand. Since it's not using the parallel interface, only one operation can happen at a time. This would be, literally, sequential execution with extra overhead, in this case . Again, in this case , what would hope to be achieved from doing things lazily, since the lazy operations would immediately be followed by their evaluation? The parallel interface, which is async, is probably what you're lookin for.
Let's look at the subtraction in this case. If evaluation is lazy, then the subtraction operator gets fed two unevaluated matrix multiplies. If it's a dumb subtraction operator, this gives us no benefit. Eventually it evaluates both and then subtracts. And it has some extra overhead like you said. But if it's a smart subtraction operator, it can realize that both parameters are the same equation, and then it can retu…
[deleted]
Re: Defeating Nondeterminism in LLM Inference
#130Earlier quoted context omitted.
I don't understand. Since it's not using the parallel interface, only one operation can happen at a time. This would be, literally, sequential execution with extra overhead, in this case . Again, in this case , what would hope to be achieved from doing things lazily, since the lazy operations would immediately be followed by their evaluation? The parallel interface, which is async, is probably what you're lookin for.
Let's look at the subtraction in this case. If evaluation is lazy, then the subtraction operator gets fed two unevaluated matrix multiplies. If it's a dumb subtraction operator, this gives us no benefit. Eventually it evaluates both and then subtracts. And it has some extra overhead like you said. But if it's a smart subtraction operator, it can realize that both parameters are the same equation, and then it can retu…
I see now, thank you. I was stuck on the "lazy evaluation" part, rather than the optimization part they were actually suggesting.