Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CompileBench: Can AI Compile 22-year-old Code?

quesma.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: CompileBench: Can AI Compile 22-year-old Code?

#51

Post by groby_b »

stared wrote: post #19
Philpax wrote: post #7

Excellent benchmark. May I suggest a extension: "port any pre-uv Python ML codebase to uv so that it can actually be reliably reproduced"?

Here the tricky part is to make tests that it work correctly. I did this upgrade a few times, and works for simple stuff like charm (e.g. removing requirements.txt and adding proper pyproject.toml). Even in Claude Code, it takes some prompting and CLAUDE.md so that it consistently runs uv, rather sometimes `uv run python`, other times `python3 -m` being surprised that some dependency is not available.

Skip the fiddling with prompts, just sandbox so that these commands are not run (via permissions.deny)

In dire cases, use the PreToolUse hook to inspect/intercept (though it's usually not necessary).

Granted, I haven't tried it for huge projects yet, but after doing that my small-medium sized projects all got ported nicely.

(If you must change the prompt, mention PEP723 as well, it seems to have the same effect as showing a shiny trinket to a magpie ;)

Re: CompileBench: Can AI Compile 22-year-old Code?

#52

Post by groby_b »

saltcured wrote: post #45

I might start to accept this LLM stuff when it can directly compile programs, i.e. not spit out a compiler command but take in source code and output linked object code in an executable format via token inference. And have it be correct. Then, I'd start to trust in its ability to manage context and reliably work through complex tasks.

"I'll try this newfangled steel constructions if they actually forge each rebar on site".

You are saying that you'd trust the new and unproven technology more if it didn't rely on old and proven technology and instead reinvented everything from scratch. That's a somewhat illogical take.

Re: CompileBench: Can AI Compile 22-year-old Code?

#53

Post by saltcured »

groby_b wrote: post #52
saltcured wrote: post #45

I might start to accept this LLM stuff when it can directly compile programs, i.e. not spit out a compiler command but take in source code and output linked object code in an executable format via token inference. And have it be correct. Then, I'd start to trust in its ability to manage context and reliably work through complex tasks.

"I'll try this newfangled steel constructions if they actually forge each rebar on site". You are saying that you'd trust the new and unproven technology more if it didn't rely on old and proven technology and instead reinvented everything from scratch. That's a somewhat illogical take.

Of course I was being facetious.

But by "correct", I meant that it would need to be able to work through such multi-level tasks as a compiler with semantic analysis, error checking, optimization, and code generation to reliably transcribe the source code. Not just emit lorum ipsum executables.

Re: CompileBench: Can AI Compile 22-year-old Code?

#54

Post by bigfishrunning »

saltcured wrote: post #45

I might start to accept this LLM stuff when it can directly compile programs, i.e. not spit out a compiler command but take in source code and output linked object code in an executable format via token inference. And have it be correct. Then, I'd start to trust in its ability to manage context and reliably work through complex tasks.

The problem is verifying the output's correctness -- you could easily end up with insidious issues like this classic: https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

Re: CompileBench: Can AI Compile 22-year-old Code?

#55

Post by magicalist »

gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

I mean this in the nicest possible way because you were just messing around on a fun thing, but...

I feel like there's a real metaphor here. 86+ people did work over two decades to maintain a cross-platform codebase and that "definitely deserves to be commended", but what "definitely felt magical" was Claude bumbling through header tweaks from compilation errors until the project compiled. And in the end what has AI wrought? A viral video but not anything to give back to the original project. Really there are multiple layers here :)

Re: CompileBench: Can AI Compile 22-year-old Code?

#56

Post by behringer »

gregsadetsky wrote: post #24
bgwalter wrote: post #23

Earlier quoted context omitted.

So essentially, you are redundant now and celebrate it.

I celebrate that I did not have to spend cycles dealing with a non-interesting, non-intellectually-challenging issue aka figuring out the incantations to make a build system happy. I'm also celebrating (although I forgot to do this - my bad!) that this automated discovery (i.e. of how to fix the build system for machines such as mine) could have been brought back to the Chocolate Doom community, and made the software…

Be aware that if you don't understand every change then your contributions may not be welcome or helpful, depending on the project and situation.

Re: CompileBench: Can AI Compile 22-year-old Code?

#58

Post by olivia-banks »

anonzzzies wrote: post #57

22 years is not that old. we run and maintain saas sites older than that.

I think it definitely depends on the software. GNU GCC coreutils/binutils are a nightmare to build if you're trying to build it on a system more than 15 years its junior.

Re: CompileBench: Can AI Compile 22-year-old Code?

#59

Post by anonzzzies »

olivia-banks wrote: post #58
anonzzzies wrote: post #57

22 years is not that old. we run and maintain saas sites older than that.

I think it definitely depends on the software. GNU GCC coreutils/binutils are a nightmare to build if you're trying to build it on a system more than 15 years its junior.

I do on ultrasparcs (5-10 and e450). But 25 year old perl/php saas is a lot easier.

Re: CompileBench: Can AI Compile 22-year-old Code?

#60

Post by hbs18 »

magicalist wrote: post #55
gregsadetsky wrote: post #22

I recently downloaded the source code for Chocolate Doom [0], and even though a ton of human labor has been put into making it cross-platform and easy to build (and that work definitely deserves to be commended!), I still couldn't build it immediately on my M1 MacBook. Asking Claude Code to build it - literally prompting it "fix whatever needs to be fixed until you get the binary to run" - and waiting ~20 minutes was…

I mean this in the nicest possible way because you were just messing around on a fun thing, but... I feel like there's a real metaphor here. 86+ people did work over two decades to maintain a cross-platform codebase and that "definitely deserves to be commended", but what "definitely felt magical" was Claude bumbling through header tweaks from compilation errors until the project compiled. And in the end what has AI…

To be fair, the topic is AI, so of course that's what he's focusing on
Post reply on HN