Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are compilers deterministic?

blog.onepatchdown.net

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Are compilers deterministic?

#61

Post by seanmcdirmid »

sarchertech wrote: post #60
seanmcdirmid wrote: post #38

Earlier quoted context omitted.

If you think that’s bad you should see how non-deterministic the alternative is (human programmers). Thankfully LLMs can iterate on the code they write, anyone who is using them to generate the same code from scratch each time a change is made needs some extra education. They are not code generators, they are junior programmers.

And junior programmers need supervision. You’re agreeing with my point though which is that LLMs aren’t higher level compilers. LLMs aren’t abstraction, they are delegation.

Yes, lots of supervision. So humans are still useful in the loop. LLMS are definitely not algorithmic transformers (ie compilers). Well, they are more like “coders” when the term meant the secretaries who translated the mathematician and scientists (the “programmers”) notation into machine code that could run on computer, by hand.

Re: Are compilers deterministic?

#62

Post by sarchertech »

IanCal wrote: post #59
sarchertech wrote: post #40

Earlier quoted context omitted.

And what happens when you change one word in the specification and the app completely changes? Sure everything you have unit tests for might stay the same, but unless your unit tests are testing all observable behavior (and if they are they’ll be 100x longer than the code) users will notice incredibly confusing differences in every build.

Vibe coding doesn't mean that it's rebuilt from the spec from scratch each time, nor does it mean nobody looks at the application. It originally meant nobody looked at the code at all, although that's much more diluted I feel now - things are called vibe coding that sit on a larger spectrum. My point was that "didn't look at the code, looked at the app and gave feature requests and tried the results" is a way of buil…

>rebuilt from scratch

I have actually encountered multiple people proposing that. People in the camp of LLMs are the new high level compiler and code is assembly language.

That’s what prompted this entire are compilers deterministic article.

But ignoring whether it’s regenerated from scratch, when you ask an agent to add minor functionality, it will regularly change existing unrelated functionality.

Some of that is accidental, and could theoretically be solved. For example agents when refactoring won’t copy and paste existing code, they will rewrite it from scratch and frequently change it. Some of it is a required because systems are intertwined.

If the application is small enough you can get away with just trying the results to build confidence. However for any non-trivial application combinatorics explosion means that this stops working very quickly. This has been understood for decades.

> My point was that "didn't look at the code, looked at the app and gave feature requests and tried the results" is a way of building applications that essentially anyone who doesn't code has been doing all along.

That is not building applications. That is delegating building applications to a trusted team of humans. If that trusted team of humans are essentially junior programmers with no agency or ability to learn and improve, your app will fail.

Imagine being a product manager with a team of 100 off shore junior developers that rotate out every day. You could 100% build something useful with that setup. But anything more complex than a todo app would eventually collapse under the chaos.

Re: Are compilers deterministic?

#63

Post by sarchertech »

seanmcdirmid wrote: post #61
sarchertech wrote: post #60

Earlier quoted context omitted.

And junior programmers need supervision. You’re agreeing with my point though which is that LLMs aren’t higher level compilers. LLMs aren’t abstraction, they are delegation.

Yes, lots of supervision. So humans are still useful in the loop. LLMS are definitely not algorithmic transformers (ie compilers). Well, they are more like “coders” when the term meant the secretaries who translated the mathematician and scientists (the “programmers”) notation into machine code that could run on computer, by hand.

The analogy I’ve found most helpful is that it’s like having access to a team of 100 offshore junior developers that rotate out completely every few hours.

There are certainly useful things you can do with that.

But give that team to a PM who can’t read the output and anything beyond a todo app would collapse under the chaos.

Re: Are compilers deterministic?

#64

Post by wat10000 »

eru wrote: post #55
wat10000 wrote: post #25

Earlier quoted context omitted.

And determinism isn’t particularly helpful with compilers. We expect adherence to some sort of spec. A compiler that emits radically different code depending on how much whitespace you put between tokens could still be completely deterministic, but it’s not the kind of tool we want to be using. Determinism is a red herring. What matters is how rigorous the relationship is between the input and the output. Compilers c…

The problem you pointed out is real, but determinism in compilers is still useful! Suppose you had one of those widely unstable compilers: concretely if you change formatting slightly, you get a totally different binary. It still does the same thing as per the language spec, but it goes about it in a completely different way. This weak determinism is still useful, because you can still get reproducible builds. Eg vol…

It’s useful but far less important. This is easily seen by the fact that we went decades building lots of software without reproducible builds.

Re: Are compilers deterministic?

#65

Post by seanmcdirmid »

sarchertech wrote: post #63
seanmcdirmid wrote: post #61

Earlier quoted context omitted.

Yes, lots of supervision. So humans are still useful in the loop. LLMS are definitely not algorithmic transformers (ie compilers). Well, they are more like “coders” when the term meant the secretaries who translated the mathematician and scientists (the “programmers”) notation into machine code that could run on computer, by hand.

The analogy I’ve found most helpful is that it’s like having access to a team of 100 offshore junior developers that rotate out completely every few hours. There are certainly useful things you can do with that. But give that team to a PM who can’t read the output and anything beyond a todo app would collapse under the chaos.

A good LLM is a junior developer who is somehow really proficient at doing unreliable research. They won’t say no, I don’t know how, like lots of junior developers, but maybe sometimes they should? They also follow instructions much better than junior developers does, and they don’t mind being micro managed.

Re: Are compilers deterministic?

#66

Post by 1718627440 »

ChadNauseam wrote: post #42
virgil_disgr4ce wrote: post #39

Earlier quoted context omitted.

> A one word change to a prompt/spec will produce a drastically different program So in other words, determinism (or lack thereof) is the hard problem!

A deterministic program must have the same output for the same input, but determinism does not restrict the outputs for different inputs

I think most people perceive program semantic not only to be defined for the whole program as a whole, but also towards individual subexpressions. If the "LLM compiler" changes the output completely when a single word is added, then it is not deterministic for the subexpressions, that don't contain that addition.

Granted that is not the rigorous definition of determinism. Also some existing compilers are non-deterministic with such a definition, namely when "undefined behaviour" would be triggered. That's precisely the problem with UB.

Re: Are compilers deterministic?

#67

Post by 1718627440 »

AlotOfReading wrote: post #45

A related property is whether particular kinds of changes to the inputs have proportionally sized changes to the output. Adding a print statement shouldn't change the behavior of the function it's in (sans I/O), for example. Using calling the same function from two different callsites shouldn't change the behavior either. A new compiler version shouldn't change the observable behavior. Etc. I think this is the more i…

Maybe it should be names with something like "algorithmic stability"? https://en.wikipedia.org/wiki/Stability_theory, https://en.wikipedia.org/wiki/Stable_theory, https://en.wikipedia.org/wiki/Lipschitz_continuity.

Re: Are compilers deterministic?

#68

Post by 1718627440 »

IanCal wrote: post #8

I’ve felt like a good response to the vibe coding thing is that customers, product managers, etc ask for features and don’t read the code. You don’t need to read the code of something to build a level of trust about what it does and whether that matches your expectations. It is not that wild that you can have a setup where you get an application and without reading the code decide if it solves your problem to your sa…

They are able to extrapolate behaviour from single examples, because the implementor has common human sense, and produces an algorithm, that is continuous, and doesn't exhibit completely different behaviour for slight input changes. That is precisely the problem here.

Re: Are compilers deterministic?

#69

Post by eru »

wat10000 wrote: post #64
eru wrote: post #55

Earlier quoted context omitted.

The problem you pointed out is real, but determinism in compilers is still useful! Suppose you had one of those widely unstable compilers: concretely if you change formatting slightly, you get a totally different binary. It still does the same thing as per the language spec, but it goes about it in a completely different way. This weak determinism is still useful, because you can still get reproducible builds. Eg vol…

It’s useful but far less important. This is easily seen by the fact that we went decades building lots of software without reproducible builds.

If you go further back in history, we went even longer without building any software!

No clue whether that's a good argument.

Re: Are compilers deterministic?

#70

Post by 1718627440 »

overgard wrote: post #50
salawat wrote: post #47

Earlier quoted context omitted.

Excuse me. I curse at compiler writer's on a regular goddamn basis. There are these things called "optimizations" that I assume they spend a bunch of time hmmming and hawwwing over, but will regularly take a shit on how the structure of how the resulting assembly comes out. It is downright infuriating. Maybe you don't build or tinker with things enough to have warranted making a dartboard out of the gcc contributor g…

While I agree undefined behavior is annoying and a huge UX issue, end of the day the bug is in your code not in the compiler. Also having an -O0 debug build is standard practice. My point isn't that compilers are super easy to use and never frustrating, my point is that the notion that LLMs "compile" english to code is a bad analogy. Compilation is a translation from one formal representation to another. LLMs are an…

Unrelated to your actual point, with which I agree with, the problem here is not just UB. It's that timing behaviour is not part of the language specification, so compilers don't necessarily uphold it, same for other machine specific semantics, like reachability of symbols or access patterns to random addresses.
Post reply on HN