Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to wrangle non-deterministic AI outputs into conventional software? (2025)

domainlanguage.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#11

Post by jkaptur »

an0malous wrote: post #5

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

(I'm not an expert. I'd love to be corrected by someone who actually knows.)

Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#12

Post by janalsncm »

an0malous wrote: post #5

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

Transformers are just a special kind of binary which are run by inference code. Where the rubber meets the road is whether the inference setup is deterministic. There’s some literature on this: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

I don’t think the issue is determinism per se but chaotic predictions that are difficult to rely on.

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#13

Post by bpodgursky »

an0malous wrote: post #5

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

Strict deterministic output for a given prompt prevents the use of RAG, which increasingly limits the relative utility of a LLM within an organization.

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#14

Post by solsane »

an0malous wrote: post #5

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

Well, you could say that about computers in general. I'm assuming you're referring to temperature (or something similar) which can be set to always pick the most probable token. Floats aside, this should be deterministic. But practically I don't think that changes much since adjusting the input slightly can lead to very different output. Also back in the day the temperature helped it avoid cyclic loops

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#15

Post by esafak »

an0malous wrote: post #5

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

The models generate a token distribution. Which one to pick is a choice. One can sample from the distribution, hence the randomness.

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#16

Post by an0malous »

solsane wrote: post #14
an0malous wrote: post #5

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

Well, you could say that about computers in general. I'm assuming you're referring to temperature (or something similar) which can be set to always pick the most probable token. Floats aside, this should be deterministic. But practically I don't think that changes much since adjusting the input slightly can lead to very different output. Also back in the day the temperature helped it avoid cyclic loops

Yes but chaotic is very different than non deterministic, and not just in an academic way because e.g. I can write tests against chaotic outputs but not really against non deterministic outputs.

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#17

Post by an0malous »

janalsncm wrote: post #12
an0malous wrote: post #5

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

Transformers are just a special kind of binary which are run by inference code. Where the rubber meets the road is whether the inference setup is deterministic. There’s some literature on this: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... I don’t think the issue is determinism per se but chaotic predictions that are difficult to rely on.

I agree they could be chaotic but I think that’s an important distinction

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#18

Post by 10000truths »

jkaptur wrote: post #11
an0malous wrote: post #5

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

(I'm not an expert. I'd love to be corrected by someone who actually knows.) Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.

Not an expert either, but my understanding is that large models use quantized weights and tensor inputs for inference. Multiplication and addition of fixed-point values is associative, so unless there's an intermediate "convert to/from IEEE float" step (activation functions, maybe?), you can still build determinism into a performant model.

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#19

Post by kimixa »

10000truths wrote: post #18
jkaptur wrote: post #11

Earlier quoted context omitted.

(I'm not an expert. I'd love to be corrected by someone who actually knows.) Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.

Not an expert either, but my understanding is that large models use quantized weights and tensor inputs for inference. Multiplication and addition of fixed-point values is associative, so unless there's an intermediate "convert to/from IEEE float" step (activation functions, maybe?), you can still build determinism into a performant model.

Fixed point arithmetic isn't truly associative unless they have infinite precision. The second you hit a limit or saturate/clamp a value the result very much depends on order of operations.

Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)

#20

Post by 10000truths »

kimixa wrote: post #19
10000truths wrote: post #18

Earlier quoted context omitted.

Not an expert either, but my understanding is that large models use quantized weights and tensor inputs for inference. Multiplication and addition of fixed-point values is associative, so unless there's an intermediate "convert to/from IEEE float" step (activation functions, maybe?), you can still build determinism into a performant model.

Fixed point arithmetic isn't truly associative unless they have infinite precision. The second you hit a limit or saturate/clamp a value the result very much depends on order of operations.

Ah yes, I forgot about saturating arithmetic. But even for that, you wouldn't need infinite precision for all values, you'd only need "enough" precision for the intermediate values, right? E.g. for an inner product of two N-element vectors containing M-bit integers, an accumulator with at least ceil(log2(N))+2*M bits would guarantee no overflow.
Post reply on HN