I have tried to get Claude to compile arbitrary C++ projects with Emscripten, and its track record is about as good as mine.
CompileBench: Can AI Compile 22-year-old Code?
11–20 of 69 posts
Re: CompileBench: Can AI Compile 22-year-old Code?
#12the libs in the bench don’t really have an external deps. will be much more interesting to see the results with ffmpeg, Qt, etc. The original source releases from any repo here would also be great candidates: https://github.com/id-software
Re: CompileBench: Can AI Compile 22-year-old Code?
#13I hadn’t thought of that use case. Say for example you find 1990’s Clipper code and want to give it a try on a modern Linux. Thanks
Re: CompileBench: Can AI Compile 22-year-old Code?
#14> Our toughest challenges include cross-compiling to Windows or ARM64 and resurrecting 22-year-old source code from 2003 on modern systems. Some agents needed 135 commands and 15 minutes just to produce a single working binary. I found that "just" there to be so funny in terms of how far the goal posts moved over these last few years (as TFA does mention). I personally am certain that it would have taken me significa…
15 minutes?
And here's me, after 4 straight days of wrangling an obscure cross-compilation toolchain to resurrect some ill-fated piece of software from year 2011 in a modern embedded environment.
Re: CompileBench: Can AI Compile 22-year-old Code?
#15Author here. So far in this benchmark we based the tasks on a couple of open-source projects (like curl, jq, GNU Coreutils). Even on those "simple" projects we managed to make the tasks difficult - Claude Opus 4.1 was the only one to correctly cross-compile curl for arm64 (+ make it statically-linked) [1]. In the future we'd like to test it with projects like FFmpeg or chromium - those should be much more difficult.…
For the _reviving 20 year old code_ type tasks, are the tested outcomes things we'd expect to be in the public domain? For example, in the way the 'SWEBenchVerified' tests are poisoned tests, because the LLMs are able to look up bug fixes in the project git repository.
Re: CompileBench: Can AI Compile 22-year-old Code?
#16[dead]
Re: CompileBench: Can AI Compile 22-year-old Code?
#17Author here. So far in this benchmark we based the tasks on a couple of open-source projects (like curl, jq, GNU Coreutils). Even on those "simple" projects we managed to make the tasks difficult - Claude Opus 4.1 was the only one to correctly cross-compile curl for arm64 (+ make it statically-linked) [1]. In the future we'd like to test it with projects like FFmpeg or chromium - those should be much more difficult.…
A long time ago, I did a project where I downloaded a year's worth of nightly builds for Thunderbird so that I could collect nightly code coverage information. Over the course of doing so, I discovered that there was one dependency (pango, I think?) such that no version could support the entire year's worth of source--the newer version didn't work with the older builds, and the older version didn't work with the newer builds.
Come to think of it, in terms of trying to get old code building, the CVS days of Firefox should be interesting... because the first command in that build step is "download the source code" and that CVS server isn't running anymore. And some of the components are downloaded from a CVS tag rather than trunk, and the converted CVS repositories I'm aware of all only converted the trunk and none of the branches or tags.
Re: CompileBench: Can AI Compile 22-year-old Code?
#18> Our toughest challenges include cross-compiling to Windows or ARM64 and resurrecting 22-year-old source code from 2003 on modern systems. Some agents needed 135 commands and 15 minutes just to produce a single working binary. I found that "just" there to be so funny in terms of how far the goal posts moved over these last few years (as TFA does mention). I personally am certain that it would have taken me significa…
Given the amount of time I've spent wrestling toolchain unpleasantness, particularly for old or embedded systems, I will happily go take a fifteen-minute coffee break while the bot does it for me.
Of course, I will probably do this with OpenAI's option, not $20 of Anthropic API credits.
Re: CompileBench: Can AI Compile 22-year-old Code?
#19Excellent benchmark. May I suggest a extension: "port any pre-uv Python ML codebase to uv so that it can actually be reliably reproduced"?
Here the tricky part is to make tests that it work correctly.
I did this upgrade a few times, and works for simple stuff like charm (e.g. removing requirements.txt and adding proper pyproject.toml).
Even in Claude Code, it takes some prompting and CLAUDE.md so that it consistently runs uv, rather sometimes `uv run python`, other times `python3 -m` being surprised that some dependency is not available.
Re: CompileBench: Can AI Compile 22-year-old Code?
#20If you asked me to do this I would want clarification on "cross-compile", "arm64" and "statically".