The real issue here is that conventional software fundamentally lacks the expressiveness to process the kind of data that LLMs can.
That’s why you’re using an LLM in the first place.
How to wrangle non-deterministic AI outputs into conventional software? (2025)
21–30 of 35 posts
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#22Earlier quoted context omitted.
Fixed point arithmetic isn't truly associative unless they have infinite precision. The second you hit a limit or saturate/clamp a value the result very much depends on order of operations.
Ah yes, I forgot about saturating arithmetic. But even for that, you wouldn't need infinite precision for all values, you'd only need "enough" precision for the intermediate values, right? E.g. for an inner product of two N-element vectors containing M-bit integers, an accumulator with at least ceil(log2(N))+2*M bits would guarantee no overflow.
True, you can increase bit width to guarantee never hit those issues, but right now saturating arithmetic on types that pretty commonly hit those values is the standard. Guaranteeing it would be a significant performance drop and/or memory use increase with current techniques to the level it would significantly affect availability and cost compared to what people expect.
Similarly you could not allow re-ordering of operations and similar - so the results are guaranteed to be deterministic (even if still "not correct" compared to infinite precision arithmetic) - but that would also have a big performance cost.
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#23Use one of these structured output libraries: https://github.com/outlines-dev/outlines https://github.com/jxnl/instructor https://github.com/guardrails-ai/guardrails https://www.askmarvin.ai/docs/text/transformation/ Some of them allow a JSON schema, others a Pydantic model (which you can transform to/from JSON).
Yeah - the author seems oblivious to that option. But to be fair - this post is more about the more basic step of choosing the schema - he argues that it is still a task for a human. In his next post https://www.domainlanguage.com/articles/context-mapping-an-a... applying some output schemas would be more useful.
By they way you don' need to use library to have output schemas: https://platform.openai.com/docs/guides/structured-outputs , https://platform.claude.com/docs/en/build-with-claude/struct...
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#24Use one of these structured output libraries: https://github.com/outlines-dev/outlines https://github.com/jxnl/instructor https://github.com/guardrails-ai/guardrails https://www.askmarvin.ai/docs/text/transformation/ Some of them allow a JSON schema, others a Pydantic model (which you can transform to/from JSON).
All those do is retry the generation if it fails and then give up.
Calling them structured output is a lie, it's structured validation.
If the LLM persists in generating bad JSON, those are useless.
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#25Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines
Strict deterministic output for a given prompt prevents the use of RAG, which increasingly limits the relative utility of a LLM within an organization.
How so? RAG is just a mechanism for querying external data sources. I don't see any need for non-determinism there.
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#26The real issue here is that conventional software fundamentally lacks the expressiveness to process the kind of data that LLMs can. That’s why you’re using an LLM in the first place.
How does this relate to the need for deterministic and consistent output?
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#27this page speaks about the same with some different approach https://selfvm.run/research
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#28Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines
(I'm not an expert. I'd love to be corrected by someone who actually knows.) Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.
I don't think the order of operations is non-deterministic between different runs. That would make programming and researching these systems more difficult than necessary.
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#29Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines
(I'm not an expert. I'd love to be corrected by someone who actually knows.) Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.
It’s usually not too difficult or expensive to avoid doing this.
Re: How to wrangle non-deterministic AI outputs into conventional software? (2025)
#30Earlier quoted context omitted.
(I'm not an expert. I'd love to be corrected by someone who actually knows.) Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.
I don't think the order of operations is non-deterministic between different runs. That would make programming and researching these systems more difficult than necessary.
It would be if you used atomics.