ProgramBench: Can language models rebuild programs from scratch?

Welcome to the future!

> Open internet with cheating detection => cheating is widespread, 20-36% of tasks are flagged for the stronger models, with source code lookup accounting for the majority of the violations.

Therefore:

> blocking internet access entirely is the appropriate default for ProgramBench

The fact that your Anthropic coding assistant has a tendency to search on the Internet code to be inserted into your program may count for an additional copyright violation (besides the possibility of reproducing recognizable fragments of its training data).

(I do not agree that copyright, at least in its current form, should be applicable to computer programs, but it is weird that the same companies who try to exploit copyrights against others also insist on the use of coding assistants that are a workaround against copyright laws, which is the main reason why they can increase programming productivity, because they may cut and paste code that you are not allowed to copy yourself.)

I would love to try this out. I have a horrible legacy project that is written in angular by a really amateur developer, full of huge blocks of copy pasted code that has minor modifications in each block. I’ve tried before to get an LLM to rewrite it to something more sensible, but I have not succeeded, usually it just ends up breaking everything. Is there a guide or some system to follow? What’s the best way to accomplish a task like this?

FWIW I think "LLMs are semideterministic" is something of a red herring. The real difference between LLM codegen and compilers is that compilers output logically the same assembly regardless of the variable names. If you're numerically solving a differential equation the compiler does not care if the floats represent heat through a pipe or dollars through a brokerage. Compilers don't care about semantic meaning, that concern is totally separated.

But even if its putatively implementing the same algorithm, LLMs certainly do not output basically the same finance Python as they would mechanical engineering Python. The style will be a little different. Sometimes the performance/clarity tradeoffs will be different. Sometimes it'll be fairly fancy and object-oriented, other times it'll be more low-level "objects are just dicts."

It's way more than a higher abstraction layer: LLM codegen involves a nontechnical tangling of concerns that doesn't exist with even the hoitiest-toitiest proof-checking compilers. It's a complete sea change. I find it incredibly disconcerting... for the same reason, by the way, that assembly programmers found Fortran and C disconcerting, and continued to reliably find employment for a good 40 years after higher-level languages were invented :) Actually even today. The assembly programmers who got hosed by C tended to be electricians who learned on the job - it's kind of cool to read old manuals from the 70s, carefully (and correctly!) explaining to electricians that a computer program is essentially an ephemeral circuit.

But I think there are specific skills around scientific thinking (learned at a formal college) and engineering carefulness (learned via hard knocks) that aren't going anywhere.

I don't think so. ProgramBench authors say no LLMs fully resolve any task, i.e. even the easiest tasks in their benchmark are unsolved. Whereas we found Opus 4.6 successfully reimplements almost every program up to gotree’s size (around 15-20 of them).

For Pkl, the preliminary results only went up to 1bn total tokens (costing $550, which would be cheap if LLMs could do the task). It might very well be solved at higher token budgets; see the report for more discussion of this.

The preliminary results are just on 4 targets. We have several Pkl-level and harder tasks in the full set which we're releasing soon.

In the following quote multiple things are not quite right:

> mostly involving higher-level languages, whereas ProgramBench are all very complex C programs (and much older programs with much more comprehensive test cases).

First, as I said above I think you're confusing the top-end of ProgramBench difficulty with the average. The quote in the OP is pretty clear that FFmpeg, SQLite, and PHP are the 3 hardest out of 200 in ProgramBench, and the bottom end is "compact CLI tools".

Second, I don't see the relevance of C vs higher-level languages, how does this make ProgramBench harder?

Third, for the test cases, I think you might be labouring under a misapprehension about how MirrorCode works? MirrorCode uses end-to-end tests from a variety of sources (the original program’s test suites, real-world data, and LLM-assisted generation). End-to-end means the stdout/stderr has to match exactly for each test case.