Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are compilers deterministic?

blog.onepatchdown.net

Post reply on HN
Print view View original

41–50 of 82 posts

Re: Are compilers deterministic?

#41

Post by ddtaylor »

virgil_disgr4ce wrote: post #39
sarchertech wrote: post #32

If we’re talking about “can we ignore the code the way we mostly ignore assembly and treat prompts as the new high level language”, determinism isn’t the hard problem. The real issue is prompt instability (chaos). A one word change to a prompt/spec will produce a drastically different program. Until that is solved there’s no world where we just check in the prompt and almost no one ever has to worry about the code.

> A one word change to a prompt/spec will produce a drastically different program So in other words, determinism (or lack thereof) is the hard problem!

Obviously they are cryptographic hashing functions as they exhibit an avalanche effect!

Re: Are compilers deterministic?

#42

Post by ChadNauseam »

virgil_disgr4ce wrote: post #39
sarchertech wrote: post #32

If we’re talking about “can we ignore the code the way we mostly ignore assembly and treat prompts as the new high level language”, determinism isn’t the hard problem. The real issue is prompt instability (chaos). A one word change to a prompt/spec will produce a drastically different program. Until that is solved there’s no world where we just check in the prompt and almost no one ever has to worry about the code.

> A one word change to a prompt/spec will produce a drastically different program So in other words, determinism (or lack thereof) is the hard problem!

A deterministic program must have the same output for the same input, but determinism does not restrict the outputs for different inputs

Re: Are compilers deterministic?

#43

Post by gaigalas »

overgard wrote: post #31

I feel like this is kind of missing the point of the argument around this. People love to say "Well you don't check your compiler output do you?" (never mind that some of us actually do for various reasons). When's the last time a compiler introduced a bug into your code? When's the last time an LLM introduced a bug into your code? There you go.

> When's the last time a compiler introduced a bug into your code?

A compiler, making my job harder by being unpredictable? All the time.

So did other programmers, users with creative input, random parallel processes running at the same time.

LLMs are actually kind of tame in comparison.

Re: Are compilers deterministic?

#44

Post by eichin »

bandrami wrote: post #28
einrealist wrote: post #17

If the output has problems, do you usually rerun the compilation with the same input (that you control)? I don't usually. What is included in the 'verify' step? Does it involve changing the generated code? If not, how do you ensure things like code quality, architectural constraints, efficiency and consistency? It's difficult, if not (economically) impossible, to write tests for these things. What if the LLM does not…

I thought to myself that I do this pretty frequently, but then I realized only if I'm going from make -j8 to make -j1. I guess parallelism does throw some indeterminancy into this

If parallelism adds indeterminacy, then you have a bug (probably in working out the dependency graph.) Not an unusual one - lots of open source in the 1990s had warnings about not building above -j1 because multi-core systems weren't that common and people weren't actually trying it themselves...

Re: Are compilers deterministic?

#45

Post by AlotOfReading »

A related property is whether particular kinds of changes to the inputs have proportionally sized changes to the output. Adding a print statement shouldn't change the behavior of the function it's in (sans I/O), for example. Using calling the same function from two different callsites shouldn't change the behavior either. A new compiler version shouldn't change the observable behavior. Etc.

I think this is the more important property and I'm not sure if it has a well-known name. The article obliquely calls it reliability, but regardless it's the key difference from LLMs. Compilers mostly achieve it, ignoring an endless list of exceptions you learn with experience.

LLMs usually don't, even with 0 temperature and floating point determinism.

Re: Are compilers deterministic?

#46

Post by roenxi »

saghm wrote: post #16
nu11ptr wrote: post #7

> The computer science answer: a compiler is deterministic as a function of its full input state. Engineering answer: most real builds do not control the full input state, so outputs drift. To me that implies the input isn't deterministic, not the compiler itself

You're not wrong but I think the point is to differentiate between the computer science "academic" answer and the engineering "pragmatic" answer. The former is concerned about correctly describing all possible behavior of the compiler, whereas the latter is concerned about what the actual experience is when using the compiler in practice. You might argue that this is redefining the question in a way that changes the…

> ... the important thing isn't the exact language but the intent behind the question ...

If we're not going to assume the input state is known then we definitely can't say what the intent behind the question is - for many engineering applications the compiler is deterministic. Debian has the whole reproducible builds thing going which has been a triumph of pragmatic engineering on a remarkable scale. And suggests that, pragmatically, compilers may be deterministic.

Re: Are compilers deterministic?

#47

Post by salawat »

overgard wrote: post #31

I feel like this is kind of missing the point of the argument around this. People love to say "Well you don't check your compiler output do you?" (never mind that some of us actually do for various reasons). When's the last time a compiler introduced a bug into your code? When's the last time an LLM introduced a bug into your code? There you go.

Excuse me. I curse at compiler writer's on a regular goddamn basis. There are these things called "optimizations" that I assume they spend a bunch of time hmmming and hawwwing over, but will regularly take a shit on how the structure of how the resulting assembly comes out. It is downright infuriating.

Maybe you don't build or tinker with things enough to have warranted making a dartboard out of the gcc contributor graph, but damnit, some of us do. That compiler is not magic, continually floats around, and when you're just trying to get something from the stage of "doesn't exist at all" to "exists", does absolutely throw curve balls your way. I start wit -O0 -g and then crank up the optimization level once everything works. Otherwise come debug time, shit's missing, stuff happens at weird times, etc. If you don't treat the compiler as spooky, you haven't paid enough attention to it.

Re: Are compilers deterministic?

#48

Post by bandrami »

eichin wrote: post #44
bandrami wrote: post #28

Earlier quoted context omitted.

I thought to myself that I do this pretty frequently, but then I realized only if I'm going from make -j8 to make -j1. I guess parallelism does throw some indeterminancy into this

If parallelism adds indeterminacy, then you have a bug (probably in working out the dependency graph.) Not an unusual one - lots of open source in the 1990s had warnings about not building above -j1 because multi-core systems weren't that common and people weren't actually trying it themselves...

Whenever I traced them, those bugs were always in the logic of the makefile rather than in the compiler. A target in fact depends on another target (generally from much earlier in the file) but the makefile doesn't specify that.

Re: Are compilers deterministic?

#49

Post by fragmede »

acedTrex wrote: post #14
IanCal wrote: post #8

I’ve felt like a good response to the vibe coding thing is that customers, product managers, etc ask for features and don’t read the code. You don’t need to read the code of something to build a level of trust about what it does and whether that matches your expectations. It is not that wild that you can have a setup where you get an application and without reading the code decide if it solves your problem to your sa…

There will absolutely be systems of the future that are entirely LLM written. Honestly they will probably be better quality than the standard offshore team output. But lets all hope these are not vital systems we end up depending on.

The failure mode, as foretold by James Cameron, is in handing control of the nuclear launch systems over to an AI. Let's all really hope not!

Re: Are compilers deterministic?

#50

Post by overgard »

salawat wrote: post #47
overgard wrote: post #31

I feel like this is kind of missing the point of the argument around this. People love to say "Well you don't check your compiler output do you?" (never mind that some of us actually do for various reasons). When's the last time a compiler introduced a bug into your code? When's the last time an LLM introduced a bug into your code? There you go.

Excuse me. I curse at compiler writer's on a regular goddamn basis. There are these things called "optimizations" that I assume they spend a bunch of time hmmming and hawwwing over, but will regularly take a shit on how the structure of how the resulting assembly comes out. It is downright infuriating. Maybe you don't build or tinker with things enough to have warranted making a dartboard out of the gcc contributor g…

While I agree undefined behavior is annoying and a huge UX issue, end of the day the bug is in your code not in the compiler.

Also having an -O0 debug build is standard practice.

My point isn't that compilers are super easy to use and never frustrating, my point is that the notion that LLMs "compile" english to code is a bad analogy. Compilation is a translation from one formal representation to another. LLMs are an interpretation of informal language into a formal language. They just are not at all the same thing.

Post reply on HN