71–80 of 82 posts
https://bazel.build/basics/hermeticity
Although I should add that a hermetic config for bazel is not something you just throw together in an afternoon. And yes, that's work you do before the first bit of source gets compiled. It's worth it though.
Earlier quoted context omitted.
It’s useful but far less important. This is easily seen by the fact that we went decades building lots of software without reproducible builds.
If you go further back in history, we went even longer without building any software! No clue whether that's a good argument.
The argument is that determinism in compilers isn't particularly important for building software because we did without it for a long time.
Your argument would be... that building software isn't particularly important for building software...?
The actual argument you'd be making would be something like, building software isn't particularly important for survival. Which is pretty obviously true, for the reason you state.
It's not uncommon to have a regression test for compilers that are written in their own language (e.g. some C compilers): compile each new version with itself, then use that to compile itself again, then use the result on unit tests or whatever, which should yield the same results as before. The point being that determinism of a particular form is expected and required in the instances where they do that. (I'm not ar…
Earlier quoted context omitted.
The analogy I’ve found most helpful is that it’s like having access to a team of 100 offshore junior developers that rotate out completely every few hours. There are certainly useful things you can do with that. But give that team to a PM who can’t read the output and anything beyond a todo app would collapse under the chaos.
A good LLM is a junior developer who is somehow really proficient at doing unreliable research. They won’t say no, I don’t know how, like lots of junior developers, but maybe sometimes they should? They also follow instructions much better than junior developers does, and they don’t mind being micro managed.
They are much much better at writing plausible looking code or providing plausible sounding answers that look correct but aren’t than any junior developer. They are much better at writing subtle bugs that won’t show up for days and will take hours to debug than any junior developer is.
You can’t mentor them into being a senior developer.
A related property is whether particular kinds of changes to the inputs have proportionally sized changes to the output. Adding a print statement shouldn't change the behavior of the function it's in (sans I/O), for example. Using calling the same function from two different callsites shouldn't change the behavior either. A new compiler version shouldn't change the observable behavior. Etc. I think this is the more i…
Earlier quoted context omitted.
A good LLM is a junior developer who is somehow really proficient at doing unreliable research. They won’t say no, I don’t know how, like lots of junior developers, but maybe sometimes they should? They also follow instructions much better than junior developers does, and they don’t mind being micro managed.
All those things are true, but there are a few things to add to it. They are much much better at writing plausible looking code or providing plausible sounding answers that look correct but aren’t than any junior developer. They are much better at writing subtle bugs that won’t show up for days and will take hours to debug than any junior developer is. You can’t mentor them into being a senior developer.
Junior developers (human) can kind of get by without testing, at least in the short term. The LLM can never get by without testing unless it’s some simple one off logic.
Compilers preserve semantics. That is part of their contract. Whether the output has instructions in one order or another does not matter as long as the output is observationally/functionally equivalent. Article does not do a good job of actually explaining this & instead meanders around sources of irrelevant "stochasticity" like timestamps & build-time UUIDs & concludes by claiming that LLMs have solved the halting…
C and C++ compilers are limited to preserving semantics for data-race free code only, though. They are allowed to turn a single load into multiple loads, or even a store into multiple stores: things that won't affect anything if you have only one thread accessing memory but for multithreaded programs, changing compiler or just making seemingly unrelated changes and recompiling can make existing data-race bugs have ef…
Earlier quoted context omitted.
All those things are true, but there are a few things to add to it. They are much much better at writing plausible looking code or providing plausible sounding answers that look correct but aren’t than any junior developer. They are much better at writing subtle bugs that won’t show up for days and will take hours to debug than any junior developer is. You can’t mentor them into being a senior developer.
Ya, but anyone who is using an LLM to write just code and isn’t also using them to write tests is nuts: they are just going to get garbage out. An LLM can totally write correct code, just not without some kind of feedback loop going on, even if that feedback loop is some form of oracle testing (where the LLM is also writing the tests, and the tests can be wrong themselves). They are also great enumerating edge cases,…
They’re also very bad at understanding what edge cases are useful to test and what edge cases can be ignored.
I had a situation just the other day where the LLM produced tests for scores of edge cases, but a managed to leave out a specific sequence that caused a very difficult to diagnose bug. I’m assuming because it decided that sequence wasn’t possible.
But it did include dozens of redundant cases that no sane human ever would have.
Earlier quoted context omitted.
> To me that implies the input isn't deterministic, not the compiler itself or the system upon which the compiler is built (as well as the compiler itself) has made some practical trade offs. the source file contents are usually deterministic. the order in which they're read and combined and build-time metadata injections often are not (and can be quite difficult to make so).
I mean, if you turn off incremental compilation and build in a container (or some other "clean room" environment), it should turn out the same each time. Local builds are very non-deterministic, but CI/CD shouldn't be. Either way it's a nitpick though, a compiler hypothetically can be deterministic, an LLM just isn't? I don't think that's even a criticism of LLMs, it's just that comparing the output of a compiler to…
lol, should. i believe you have to control the clock as well and even then non-determinism can still be introduced by scheduler noise. maybe it's better now, but it used to be very painful.
> Either way it's a nitpick though, a compiler hypothetically can be deterministic, an LLM just isn't? I don't think that's even a criticism of LLMs, it's just that comparing the output of a compiler to the output of an LLM is a bad analogy.
llm inference is literally sampling a distribution. the core distinction is real though, llms are stochastic general computation where traditional programming is deterministic in spirit. llm inference can hypothetically be deterministic as well if you use a fixed seed, although, like non-trivial software builds on modern operating systems, squeezing out all the entropy is a non-trivial affair. (some research labs are focused on just that, deterministic llm inference.)
Earlier quoted context omitted.
Ya, but anyone who is using an LLM to write just code and isn’t also using them to write tests is nuts: they are just going to get garbage out. An LLM can totally write correct code, just not without some kind of feedback loop going on, even if that feedback loop is some form of oracle testing (where the LLM is also writing the tests, and the tests can be wrong themselves). They are also great enumerating edge cases,…
It’s definitely true that LLMs require tests, but tests aren’t an antidote to what I’m taking about because LLMs are also good at writing plausible looking tests that are actually terrible. They’re also very bad at understanding what edge cases are useful to test and what edge cases can be ignored. I had a situation just the other day where the LLM produced tests for scores of edge cases, but a managed to leave out a…
I just went through this a couple of weeks ago. My test cases were ok but missing some basic edge cases. I asked it to enumerate the edge cases but that didn’t work well either. I then asked it to list near positive and near negative edge cases, and that worked well for my problem. For bugs I found in the code that weren’t covered, I asked the LLM to add them and/or mark them critical to cover (the case might had been there already but I only let it generate 3 tests with a limited size data set, so not all cases would be covered). That worked well, although I’m sure I could do better (probably by generating more tests, but many current approach is slow because each artifact is generated in a context isolated agent, so I have to speed that up first).