Nobody Reviews Compiler Output

When optimizing code it's not unusual to look at the assembly. It's not unusual to look for opportunities for autovectorization or to verify inlining or loop unrolling.

Compilers are, for the most part, deterministic. This means after people have reviewed the output, it's unlikely to change. It also means if they do change, only a few people are required to notice.

None of this applies to LLMs. They are worse than compilers, in regards to the quality and characteristics of their output, in every possible way.

If no one reviewed compiler output then https://godbolt.org/ wouldn't exist.

No. In reality, this is almost never done anymore.

We used to do it all the time back when performance mattered, but that was then.

HN readers don't have to like it, and obviously they (we) don't, but shooting the messenger won't help.

“AI-checks-AI pipelines as first-class CI infrastructure, not bolt-on curiosity”—what’s the contrast here? Is it serious aspiration, not unserious aspiration?

“Formal specification layers that agents execute against, not just prompts”—Okay.

It just looks like it is stating lots of problems with a x-not-y as if there is progress being made by way of insistence.

I am open to the idea of something like a small verification kernel that can be comprehended by “humans” which can check GenAI output. But right now we can contrast mature (decade+) compilers with GenAI like this.

- Compilers: You get the abstraction you asked for: it might not be “optimal” code, but it is code that works the way you wrote it

- GenAI: Here is 200KLOC, good luck, could be anything

Now you could reduce the space of those 200KLOC with tests and verification. But so far (based on this submission) it looks like this is at the handwaving stage.

Certainly you would need high-value tests if tests are the thing that is supposed to be the verification. Either something simple and expressive enough for “humans” to write or something that is both short and easy to read for “humans” (and generated by GenAI). Not some copy-paste smelling mockfest that looks like it is a pile of junk that has evolved over five years, each author pushing some junk on top while taking care to not make the whole pile tilt and collapse.

This is so dumb.