Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CompileBench: Can AI Compile 22-year-old Code?

quesma.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: CompileBench: Can AI Compile 22-year-old Code?

#31

Post by sujee_dev »

stared wrote: post #19
Philpax wrote: post #7

Excellent benchmark. May I suggest a extension: "port any pre-uv Python ML codebase to uv so that it can actually be reliably reproduced"?

Here the tricky part is to make tests that it work correctly. I did this upgrade a few times, and works for simple stuff like charm (e.g. removing requirements.txt and adding proper pyproject.toml). Even in Claude Code, it takes some prompting and CLAUDE.md so that it consistently runs uv, rather sometimes `uv run python`, other times `python3 -m` being surprised that some dependency is not available.

I am doing this now. What are your instructions in CLAUDE.md? thx

Re: CompileBench: Can AI Compile 22-year-old Code?

#32

Post by fuhsnn »

For C projects, the task should be passing the full test suite with at least address-sanitizer enabled. Amusing how some would discourage fellow human from using a programming language because of its unsafeness or undefined behavior, yet AI doing unaudited source modification on the same language is encouraged.

Re: CompileBench: Can AI Compile 22-year-old Code?

#33

Post by peatmoss »

Though this is more "LLM uses a variety of open source tools and compilers to compile source," I do wonder about whether there will eventually be a role for transformers in compiling code.

I've mentioned this before, but "sufficiently smart compiler" would be the dream here. Start with high level code or pseudo code, end up with something optimized.

Re: CompileBench: Can AI Compile 22-year-old Code?

#34

Post by fuhsnn »

piotrgrabowski wrote: post #3

Author here. So far in this benchmark we based the tasks on a couple of open-source projects (like curl, jq, GNU Coreutils). Even on those "simple" projects we managed to make the tasks difficult - Claude Opus 4.1 was the only one to correctly cross-compile curl for arm64 (+ make it statically-linked) [1]. In the future we'd like to test it with projects like FFmpeg or chromium - those should be much more difficult.…

You didn't make the tasks difficult, you make them easier.

The entire coreutils is reduced to one utility (sha1sum) and the test doesn't even try to feed a real file to it (just a stdin string)[0], same goes to the jq task, there isn't even a json file feed to it, what's being verified[1] is barely a calculator.

These project ship with "make check", please tell AI to use it.

[0] https://github.com/QuesmaOrg/CompileBench/blob/86d9aeda88a16...

[1] https://github.com/QuesmaOrg/CompileBench/blob/86d9aeda88a16...

Re: CompileBench: Can AI Compile 22-year-old Code?

#35

Post by jstummbillig »

bgwalter wrote: post #23
gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

So essentially, you are redundant now and celebrate it.

Naturally, the primary source of purpose in life: Making Chocolate Doom compile.

Re: CompileBench: Can AI Compile 22-year-old Code?

#36

Post by qazxcvbnmlp »

ACCount37 wrote: post #14
falcor84 wrote: post #6

> Our toughest challenges include cross-compiling to Windows or ARM64 and resurrecting 22-year-old source code from 2003 on modern systems. Some agents needed 135 commands and 15 minutes just to produce a single working binary. I found that "just" there to be so funny in terms of how far the goal posts moved over these last few years (as TFA does mention). I personally am certain that it would have taken me significa…

15 minutes? And here's me, after 4 straight days of wrangling an obscure cross-compilation toolchain to resurrect some ill-fated piece of software from year 2011 in a modern embedded environment.

Letting an agent figure out how to compile old projects is magical. What used to be multiple days of slog is now “compile this, make changes and download tools as needed” with 10 mins of git review to make sure it didn’t do anything stupid.

Re: CompileBench: Can AI Compile 22-year-old Code?

#37

Post by calebkaiser »

peatmoss wrote: post #33

Though this is more "LLM uses a variety of open source tools and compilers to compile source," I do wonder about whether there will eventually be a role for transformers in compiling code. I've mentioned this before, but "sufficiently smart compiler" would be the dream here. Start with high level code or pseudo code, end up with something optimized.

There's been a decent chunk of research in this direction over the years. Michael O'Boyle is pretty active as a researcher in the space, if you're looking for stuff to read: https://www.dcs.ed.ac.uk/home/mob/

Re: CompileBench: Can AI Compile 22-year-old Code?

#38

Post by a456463 »

bgwalter wrote: post #26
varispeed wrote: post #25

Earlier quoted context omitted.

It's like saying chisel made carpenter redundant. AI still needs an operator and then more people to actually make the output production ready.

People here are claiming that "AI" emits fully working products, so with that reading they are not just a tool. Also, you would own a chisel and the chisel does not spy on you. The "AI" factories are owned by oligopolies and you have to pay a steep monthly fee in order to continue receiving your warez that are derivative works of actually creative people's IP. Also, the "AI" factories know everything you do and ask a…

You are correct and I agree with you. HN monoculture of AI fanbois won't understand this

Re: CompileBench: Can AI Compile 22-year-old Code?

#39

Post by a456463 »

bgwalter wrote: post #23
gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

So essentially, you are redundant now and celebrate it.

Precisely

Re: CompileBench: Can AI Compile 22-year-old Code?

#40

Post by simonw »

bgwalter wrote: post #26
varispeed wrote: post #25

Earlier quoted context omitted.

It's like saying chisel made carpenter redundant. AI still needs an operator and then more people to actually make the output production ready.

People here are claiming that "AI" emits fully working products, so with that reading they are not just a tool. Also, you would own a chisel and the chisel does not spy on you. The "AI" factories are owned by oligopolies and you have to pay a steep monthly fee in order to continue receiving your warez that are derivative works of actually creative people's IP. Also, the "AI" factories know everything you do and ask a…

I think you're in the wrong thread. This isn't about AI emitting "fully working products", this is about AI brute-force figuring out how to compile stuff with gnarly constraints, a task which very few software developers look forward to.

Plus, as other commenters have pointed out already, you can run this stuff entirely free from risk of an AI company spying on what you are doing. The models that run locally got really good in the past 12 months, and if they don't work on your own machine you can rent a capable cloud GPU machine for a few bucks an hour.

Post reply on HN