51–60 of 82 posts
> The computer science answer: a compiler is deterministic as a function of its full input state. Engineering answer: most real builds do not control the full input state, so outputs drift. To me that implies the input isn't deterministic, not the compiler itself
> The computer science answer: a compiler is deterministic as a function of its full input state. Engineering answer: most real builds do not control the full input state, so outputs drift. To me that implies the input isn't deterministic, not the compiler itself
> To me that implies the input isn't deterministic, not the compiler itself or the system upon which the compiler is built (as well as the compiler itself) has made some practical trade offs. the source file contents are usually deterministic. the order in which they're read and combined and build-time metadata injections often are not (and can be quite difficult to make so).
Either way it's a nitpick though, a compiler hypothetically can be deterministic, an LLM just isn't? I don't think that's even a criticism of LLMs, it's just that comparing the output of a compiler to the output of an LLM is a bad analogy.
> . I’m AI-pilled enough to daily-drive comma.ai, and I still want deterministic verification gates around generated code. My girlfriend prefers when I let it drive because it’s smoother and less erratic than I am, which is a useful reminder that “probabilistic system” and “operationally better result” can coexist. When did the girlfriend enter the discussion? Did I miss something?
The OP brings up testimony of someone other than himself who prefers when software drives their car rather than him.
> This comes up now as “is vibecoding sane if LLMs are nondeterministic?” Again: do you want the CS answer, or the engineering answer? Determinism would help you. With a bit of engineering, you could make LLMs deterministic: basically, fix the random seed for the PRNG and make sure none of the other sources of entropy mentioned earlier in the article contribute. But that barely impact any of the issues people bring u…
If you've got a fixed GPU that doesn't degrade at all during the process, I think? If you switch GPUs (even another one of the same model) or run it long enough the feed-forward of rounding will produce different results, right?
> This comes up now as “is vibecoding sane if LLMs are nondeterministic?” Again: do you want the CS answer, or the engineering answer? Determinism would help you. With a bit of engineering, you could make LLMs deterministic: basically, fix the random seed for the PRNG and make sure none of the other sources of entropy mentioned earlier in the article contribute. But that barely impact any of the issues people bring u…
And determinism isn’t particularly helpful with compilers. We expect adherence to some sort of spec. A compiler that emits radically different code depending on how much whitespace you put between tokens could still be completely deterministic, but it’s not the kind of tool we want to be using. Determinism is a red herring. What matters is how rigorous the relationship is between the input and the output. Compilers c…
Suppose you had one of those widely unstable compilers: concretely if you change formatting slightly, you get a totally different binary. It still does the same thing as per the language spec, but it goes about it in a completely different way.
This weak determinism is still useful, because you can still get reproducible builds. Eg volunteers can still audit eg debian binary packages by just re-running the compiler with the exact same input to check that the output matches. So they can verify that no supply chain attack has fiddled with the binaries: at least the binaries belong to the sources the are claimed to.
Earlier quoted context omitted.
C and C++ compilers are limited to preserving semantics for data-race free code only, though. They are allowed to turn a single load into multiple loads, or even a store into multiple stores: things that won't affect anything if you have only one thread accessing memory but for multithreaded programs, changing compiler or just making seemingly unrelated changes and recompiling can make existing data-race bugs have ef…
Data races are undefined behavior¹ so in that case the compiler is still technically preserving semantics but if you use the proper primitives to remove undefined behavior (atomic operations, locks) for any shared state then the compiler will not generate code w/ undefined behavior & all modifications/mutations will be serialized in some order. You can then further refine the code if you want the operations to happen…
Earlier quoted context omitted.
If you've got a fixed GPU that doesn't degrade at all during the process, I think? If you switch GPUs (even another one of the same model) or run it long enough the feed-forward of rounding will produce different results, right?
Why would rounding not be deterministic?
> The computer science answer: a compiler is deterministic as a function of its full input state. Engineering answer: most real builds do not control the full input state, so outputs drift. To me that implies the input isn't deterministic, not the compiler itself
You're not wrong but I think the point is to differentiate between the computer science "academic" answer and the engineering "pragmatic" answer. The former is concerned about correctly describing all possible behavior of the compiler, whereas the latter is concerned about what the actual experience is when using the compiler in practice. You might argue that this is redefining the question in a way that changes the…
I’ve felt like a good response to the vibe coding thing is that customers, product managers, etc ask for features and don’t read the code. You don’t need to read the code of something to build a level of trust about what it does and whether that matches your expectations. It is not that wild that you can have a setup where you get an application and without reading the code decide if it solves your problem to your sa…
And what happens when you change one word in the specification and the app completely changes? Sure everything you have unit tests for might stay the same, but unless your unit tests are testing all observable behavior (and if they are they’ll be 100x longer than the code) users will notice incredibly confusing differences in every build.
My point was that "didn't look at the code, looked at the app and gave feature requests and tried the results" is a way of building applications that essentially anyone who doesn't code has been doing all along.
Earlier quoted context omitted.
It’s much worse than that. LLMs today don’t produce the same output for the same prompt.
If you think that’s bad you should see how non-deterministic the alternative is (human programmers). Thankfully LLMs can iterate on the code they write, anyone who is using them to generate the same code from scratch each time a change is made needs some extra education. They are not code generators, they are junior programmers.
You’re agreeing with my point though which is that LLMs aren’t higher level compilers. LLMs aren’t abstraction, they are delegation.