CompileBench: Can AI Compile 22-year-old Code?

#1

Post by jakozaur » Mon, Sep 22, 2025, 12:59 PM UTC

CompileBench: Can AI Compile 22-year-old Code?
quesma.com

Re: CompileBench: Can AI Compile 22-year-old Code?

#2

Post by stared » Mon, Sep 22, 2025, 1:26 PM UTC

Curious for the ultimate benchmark - can AI compile Doom an on arbitrary device?

Re: CompileBench: Can AI Compile 22-year-old Code?

#3

Post by piotrgrabowski » Mon, Sep 22, 2025, 1:57 PM UTC

Author here.

So far in this benchmark we based the tasks on a couple of open-source projects (like curl, jq, GNU Coreutils).

Even on those "simple" projects we managed to make the tasks difficult - Claude Opus 4.1 was the only one to correctly cross-compile curl for arm64 (+ make it statically-linked) [1].

In the future we'd like to test it with projects like FFmpeg or chromium - those should be much more difficult.

[1] https://www.compilebench.com/curl-ssl-arm64-static/

Re: CompileBench: Can AI Compile 22-year-old Code?

#4

Post by nl » Mon, Sep 22, 2025, 1:58 PM UTC

This is a really good benchmark. So much time is spent on these messy types of tasks and no one really likes doing it.

Now if it could fix React Native builds after package upgrades I'd be impressed...

Re: CompileBench: Can AI Compile 22-year-old Code?

#5

Post by bgwalter » Mon, Sep 22, 2025, 2:01 PM UTC

LGTM! I'm sure it comes with a correctness proof, too!

The newer blog posts appear to scan forums like this one for objections ("AI" does not work for legacy code bases) and then create custom "benchmarks" for their sales people to point to if they encounter these objections.

Re: CompileBench: Can AI Compile 22-year-old Code?

#6

Post by falcor84 » Mon, Sep 22, 2025, 2:13 PM UTC

> Our toughest challenges include cross-compiling to Windows or ARM64 and resurrecting 22-year-old source code from 2003 on modern systems. Some agents needed 135 commands and 15 minutes just to produce a single working binary.

I found that "just" there to be so funny in terms of how far the goal posts moved over these last few years (as TFA does mention). I personally am certain that it would have taken me significantly longer than that to do it myself.

Re: CompileBench: Can AI Compile 22-year-old Code?

#7

Post by Philpax » Mon, Sep 22, 2025, 2:21 PM UTC

Excellent benchmark. May I suggest a extension: "port any pre-uv Python ML codebase to uv so that it can actually be reliably reproduced"?

Re: CompileBench: Can AI Compile 22-year-old Code?

#8

Post by buildbot » Mon, Sep 22, 2025, 2:28 PM UTC

I’ve been doing this a lot! AI seems to really excel at setting up compiler boilerplate/minor modifications for new arch. I made a simple cpu information utility work on HP PA-RISC and Sparc64 :)

Re: CompileBench: Can AI Compile 22-year-old Code?

#9

Post by flenserboy » Mon, Sep 22, 2025, 2:32 PM UTC

stared wrote: post #2

Curious for the ultimate benchmark - can AI compile Doom an on arbitrary device?

that, & how well does it cope with Perl?

Re: CompileBench: Can AI Compile 22-year-old Code?

#10

Post by johnisgood » Mon, Sep 22, 2025, 2:36 PM UTC

flenserboy wrote: post #9

stared wrote: post #2

Curious for the ultimate benchmark - can AI compile Doom an on arbitrary device?

that, & how well does it cope with Perl?

Claude is good enough at Perl with lots of hand-holding and reiterations, according to my experiences.