I feel like this is kind of missing the point of the argument around this. People love to say "Well you don't check your compiler output do you?" (never mind that some of us actually do for various reasons). When's the last time a compiler introduced a bug into your code? When's the last time an LLM introduced a bug into your code? There you go.
Are compilers deterministic?
31–40 of 82 posts
Re: Are compilers deterministic?
#32If we’re talking about “can we ignore the code the way we mostly ignore assembly and treat prompts as the new high level language”, determinism isn’t the hard problem.
The real issue is prompt instability (chaos). A one word change to a prompt/spec will produce a drastically different program. Until that is solved there’s no world where we just check in the prompt and almost no one ever has to worry about the code.
Re: Are compilers deterministic?
#33Yes, yes they are. Lots of engineering effort goes into making this be true. TFA argues that you can't control the inputs perfectly, and so the behavior may differ if you fail to control the inputs. Yeah sure. But the answer to the clickbaity question in the title is simply "Yes".
Even the specific example in the article, the non-determinism was treated as a bug and was fixed (since by 2004 that was definitely a regression - we put a lot of work in, in the mid to late 1990s, to get bit level reproducibility - and even before that, those little details like timestamps were still deterministic variations, we had binary diff tools that could filter them out.)
Re: Are compilers deterministic?
#34Compilers preserve semantics. That is part of their contract. Whether the output has instructions in one order or another does not matter as long as the output is observationally/functionally equivalent. Article does not do a good job of actually explaining this & instead meanders around sources of irrelevant "stochasticity" like timestamps & build-time UUIDs & concludes by claiming that LLMs have solved the halting…
C and C++ compilers are limited to preserving semantics for data-race free code only, though. They are allowed to turn a single load into multiple loads, or even a store into multiple stores: things that won't affect anything if you have only one thread accessing memory but for multithreaded programs, changing compiler or just making seemingly unrelated changes and recompiling can make existing data-race bugs have ef…
Data races are undefined behavior¹ so in that case the compiler is still technically preserving semantics but if you use the proper primitives to remove undefined behavior (atomic operations, locks) for any shared state then the compiler will not generate code w/ undefined behavior & all modifications/mutations will be serialized in some order. You can then further refine the code if you want the operations to happen in a certain order. At the end of the day you must assume that the compiler will preserve your intended semantics otherwise we'd still be writing assembly b/c no high level specification would ever mean what it was intended to mean for the low-level executable machine model/target of the compiler.
Re: Are compilers deterministic?
#35I’ve felt like a good response to the vibe coding thing is that customers, product managers, etc ask for features and don’t read the code. You don’t need to read the code of something to build a level of trust about what it does and whether that matches your expectations. It is not that wild that you can have a setup where you get an application and without reading the code decide if it solves your problem to your sa…
I'm using Claude every day and I am correcting the output all the time, by reading the code, because it does bad things and breaks things. I feel like the people saying that it doesn't matter either aren't building anything real or they're outsourcing their work onto reviewers or they have a time bomb on their hands.
Re: Are compilers deterministic?
#36If we’re talking about “can we ignore the code the way we mostly ignore assembly and treat prompts as the new high level language”, determinism isn’t the hard problem. The real issue is prompt instability (chaos). A one word change to a prompt/spec will produce a drastically different program. Until that is solved there’s no world where we just check in the prompt and almost no one ever has to worry about the code.
It’s much worse than that. LLMs today don’t produce the same output for the same prompt.
Re: Are compilers deterministic?
#37> This comes up now as “is vibecoding sane if LLMs are nondeterministic?” Again: do you want the CS answer, or the engineering answer? Determinism would help you. With a bit of engineering, you could make LLMs deterministic: basically, fix the random seed for the PRNG and make sure none of the other sources of entropy mentioned earlier in the article contribute. But that barely impact any of the issues people bring u…
And determinism isn’t particularly helpful with compilers. We expect adherence to some sort of spec. A compiler that emits radically different code depending on how much whitespace you put between tokens could still be completely deterministic, but it’s not the kind of tool we want to be using. Determinism is a red herring. What matters is how rigorous the relationship is between the input and the output. Compilers c…
And the reason that relationship can be regiorous is because compilers by definition translate one formal language to another. You can’t have a compiler that translates English to machine code in a rigorous, repeatable manner because English is ambiguous.
Re: Are compilers deterministic?
#38If we’re talking about “can we ignore the code the way we mostly ignore assembly and treat prompts as the new high level language”, determinism isn’t the hard problem. The real issue is prompt instability (chaos). A one word change to a prompt/spec will produce a drastically different program. Until that is solved there’s no world where we just check in the prompt and almost no one ever has to worry about the code.
It’s much worse than that. LLMs today don’t produce the same output for the same prompt.
If you think that’s bad you should see how non-deterministic the alternative is (human programmers). Thankfully LLMs can iterate on the code they write, anyone who is using them to generate the same code from scratch each time a change is made needs some extra education. They are not code generators, they are junior programmers.
Re: Are compilers deterministic?
#39If we’re talking about “can we ignore the code the way we mostly ignore assembly and treat prompts as the new high level language”, determinism isn’t the hard problem. The real issue is prompt instability (chaos). A one word change to a prompt/spec will produce a drastically different program. Until that is solved there’s no world where we just check in the prompt and almost no one ever has to worry about the code.
> A one word change to a prompt/spec will produce a drastically different program
So in other words, determinism (or lack thereof) is the hard problem!
Re: Are compilers deterministic?
#40I’ve felt like a good response to the vibe coding thing is that customers, product managers, etc ask for features and don’t read the code. You don’t need to read the code of something to build a level of trust about what it does and whether that matches your expectations. It is not that wild that you can have a setup where you get an application and without reading the code decide if it solves your problem to your sa…
And what happens when you change one word in the specification and the app completely changes?
Sure everything you have unit tests for might stay the same, but unless your unit tests are testing all observable behavior (and if they are they’ll be 100x longer than the code) users will notice incredibly confusing differences in every build.