Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CompileBench: Can AI Compile 22-year-old Code?

quesma.com

Post reply on HN
Print view View original

21–30 of 69 posts

Re: CompileBench: Can AI Compile 22-year-old Code?

#21

Post by Palomides »

falcor84 wrote: post #6

> Our toughest challenges include cross-compiling to Windows or ARM64 and resurrecting 22-year-old source code from 2003 on modern systems. Some agents needed 135 commands and 15 minutes just to produce a single working binary. I found that "just" there to be so funny in terms of how far the goal posts moved over these last few years (as TFA does mention). I personally am certain that it would have taken me significa…

man I dunno, I was expecting some magic but the tasks seem to boil down to untar, configure with some flags, make install

it does seem the machine is faster than me since I would have to spend a minute to copy each of the --disable-whatever flags for curl

it's somewhat cool to see a computer can do the same half-assed process I do of seeing what linker failures happen and googling for the missing lib flag

Re: CompileBench: Can AI Compile 22-year-old Code?

#22

Post by gregsadetsky »

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook.

Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was the best investment of non-time I could do... It definitely felt magical. Claude would tweak headers, `make` it, try to run it, and apply more fixes based on the errors it got back.

Now that I think of it, I regret not opening an issue/PR with its findings...!

(((I then went on to make more vibe-changes to the Doom code and made a video out of those which went semi-viral, which I will now unashamedly plug [1])))

[0] https://github.com/chocolate-doom/chocolate-doom

[1] https://www.youtube.com/watch?v=LcnBXtttF28

Re: CompileBench: Can AI Compile 22-year-old Code?

#23

Post by bgwalter »

gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

So essentially, you are redundant now and celebrate it.

Re: CompileBench: Can AI Compile 22-year-old Code?

#24

Post by gregsadetsky »

bgwalter wrote: post #23
gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

So essentially, you are redundant now and celebrate it.

I celebrate that I did not have to spend cycles dealing with a non-interesting, non-intellectually-challenging issue aka figuring out the incantations to make a build system happy.

I'm also celebrating (although I forgot to do this - my bad!) that this automated discovery (i.e. of how to fix the build system for machines such as mine) could have been brought back to the Chocolate Doom community, and made the software better for everyone.

And finally, I'm also celebrating that this allowed my (if I may speak so boldly) creativity to express itself by helping me quickly bring a funny idea to life and share it, hopefully entertaining the world/making at least one person laugh/chuckle.

I don't see how any of this makes me redundant though. Efficient? Lazy? Both? Neither? But not redundant. I think! :-)

Re: CompileBench: Can AI Compile 22-year-old Code?

#25

Post by varispeed »

bgwalter wrote: post #23
gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

So essentially, you are redundant now and celebrate it.

It's like saying chisel made carpenter redundant. AI still needs an operator and then more people to actually make the output production ready.

Re: CompileBench: Can AI Compile 22-year-old Code?

#26

Post by bgwalter »

varispeed wrote: post #25
bgwalter wrote: post #23

Earlier quoted context omitted.

So essentially, you are redundant now and celebrate it.

It's like saying chisel made carpenter redundant. AI still needs an operator and then more people to actually make the output production ready.

People here are claiming that "AI" emits fully working products, so with that reading they are not just a tool.

Also, you would own a chisel and the chisel does not spy on you. The "AI" factories are owned by oligopolies and you have to pay a steep monthly fee in order to continue receiving your warez that are derivative works of actually creative people's IP. Also, the "AI" factories know everything you do and ask and what kind of code you write.

Re: CompileBench: Can AI Compile 22-year-old Code?

#27

Post by chickenzzzzu »

bgwalter wrote: post #23
gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

So essentially, you are redundant now and celebrate it.

[dead]

Re: CompileBench: Can AI Compile 22-year-old Code?

#28

Post by criemen »

OtherShrezzing wrote: post #15
piotrgrabowski wrote: post #3

Author here. So far in this benchmark we based the tasks on a couple of open-source projects (like curl, jq, GNU Coreutils). Even on those "simple" projects we managed to make the tasks difficult - Claude Opus 4.1 was the only one to correctly cross-compile curl for arm64 (+ make it statically-linked) [1]. In the future we'd like to test it with projects like FFmpeg or chromium - those should be much more difficult.…

For the _reviving 20 year old code_ type tasks, are the tested outcomes things we'd expect to be in the public domain? For example, in the way the 'SWEBenchVerified' tests are poisoned tests, because the LLMs are able to look up bug fixes in the project git repository.

> because the LLMs are able to look up bug fixes in the project git repository

That's not the (only) problem: Even if you take the internet away, we know/assume that all LLMs are heavily trained on public GitHub repositories. Therefore, they know/remember details of the code and organization in a way they can't for your private (or new, past knowledge cut-off date) code.

Re: CompileBench: Can AI Compile 22-year-old Code?

#29

Post by bongodongobob »

bgwalter wrote: post #23
gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

So essentially, you are redundant now and celebrate it.

Unless you are selling a service to compile things for people I'm not sure who is being made redundant here.

Re: CompileBench: Can AI Compile 22-year-old Code?

#30

Post by bckr »

bgwalter wrote: post #26
varispeed wrote: post #25

Earlier quoted context omitted.

It's like saying chisel made carpenter redundant. AI still needs an operator and then more people to actually make the output production ready.

People here are claiming that "AI" emits fully working products, so with that reading they are not just a tool. Also, you would own a chisel and the chisel does not spy on you. The "AI" factories are owned by oligopolies and you have to pay a steep monthly fee in order to continue receiving your warez that are derivative works of actually creative people's IP. Also, the "AI" factories know everything you do and ask a…

For now. Open Source AI continues to make progress
Post reply on HN