Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ProgramBench: Can language models rebuild programs from scratch?

arxiv.org

Post reply on HN
Print view View original

51–60 of 86 posts

Re: ProgramBench: Can language models rebuild programs from scratch?

#51

Post by Zigurd »

stingraycharles wrote: post #37
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

Problem with these types of benchmarks is that it’s 100% certain the LLM has been trained on all that code already, so they’re all tainted since you don’t know whether it’s just benchmarking recall vs actual reasoning. Same with SWE-bench and others.

That's a feature not a bug. It doesn't make benchmarking any more meaningful or simple, but being trained to recall patterns is a legitimate goal for a coding agent.

Re: ProgramBench: Can language models rebuild programs from scratch?

#52

Post by LeCompteSftware »

tadamcz wrote: post #44
LeCompteSftware wrote: post #35

Earlier quoted context omitted.

Surely the biggest difference is that you guys are mostly testing LLMs on simpler utilities, mostly involving higher-level languages, whereas ProgramBench are all very complex C programs (and much older programs with much more comprehensive test cases). Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. In fact the only program you tested which actually has anywhere cl…

> Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. This is incidental to the main disagreement, but btw I also doubt this. Let's try and make the claim more precise. e.g. are you saying the average university undergraduate studying CS would reimplement cal from scratch (only stdlib), matching the output perfectly for all 1365 MirrorCode test cases, in (say) 3 days of…

I didn't say "3 days of full-time work," that is totally unreasonable. I was giving them basically unlimited time to do whatever slow testing and research they needed. And let me qualify my statement: when I say "I would expect most sophomores to be able to do this," I mean "if most sophomores can't do this then their university is badly failing them." (If you want to split hairs about modern undergrads not learning C then I think this conversation is over.)

Of course it would take them a while to learn facts about datetime that the LLM doesn't need to learn. If your argument is about cost optimization then congrats, you win. The point is that it doesn't take a huge amount of C expertise to do this successfully - the standard implementation is nothing you wouldn't see in K&R: https://raw.githubusercontent.com/util-linux/util-linux/refs... It's routine.

But a nontrivial database, even a simple one like SQLite, really does require professional-level C expertise. It is not routine. So your comparison to ProgramBench still seems apple-to-oranges.

Re: ProgramBench: Can language models rebuild programs from scratch?

#53

Post by killerstorm »

It's a very misleading: they don't provide any meaningful documentation/requirements. Just an executable blackbox.

E.g. a doc for ffmpeg, which I checked by downloading docker image they provide to the model, is a README which basically just says this is ffmpeg and docs can be found online. They do not allow models to get online.

So a model is supposed to reverse-engineer a blackbox using only limited number of tries. I'm not sure even ASI can do this under these constraints (without memorizing the ffmpeg code base, obviously.)

In the only posts one of authors mentions "usage docs". Obviously they had a command-line tool like `grep` in mind -- where a man page sort-of specifies program behavior. But then added sqlite, ffmpeg, php, etc. - where a usage doc is like one millionth of information you need to implement ffmpeg.

And, of course, there's no human baseline. I'd guess making such a baseline would cost billions of dollars.

Re: ProgramBench: Can language models rebuild programs from scratch?

#54

Post by thomashop »

killerstorm wrote: post #53

It's a very misleading: they don't provide any meaningful documentation/requirements. Just an executable blackbox. E.g. a doc for ffmpeg, which I checked by downloading docker image they provide to the model, is a README which basically just says this is ffmpeg and docs can be found online. They do not allow models to get online. So a model is supposed to reverse-engineer a blackbox using only limited number of tries…

i thought the agent can execute real ffmpeg to compare

Re: ProgramBench: Can language models rebuild programs from scratch?

#55

Post by killerstorm »

thomashop wrote: post #54
killerstorm wrote: post #53

It's a very misleading: they don't provide any meaningful documentation/requirements. Just an executable blackbox. E.g. a doc for ffmpeg, which I checked by downloading docker image they provide to the model, is a README which basically just says this is ffmpeg and docs can be found online. They do not allow models to get online. So a model is supposed to reverse-engineer a blackbox using only limited number of tries…

i thought the agent can execute real ffmpeg to compare

I think you underestimate complexity of audio & video encoding standards. There are hundreds and hundreds of pages of specification. How many times do you need to execute real ffmpeg to get all tiny details?

It's certainly possible to reverse-engineer it from a blackbox access, but it would take *years* and this test has a time limit.

Re: ProgramBench: Can language models rebuild programs from scratch?

#57

Post by jerf »

tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

"But I suspect that the ProgramBench authors are either under-eliciting the AIs, or their tasks are unfair/impossible given the constraints, or both."

I'd go with "impossible":

"Given a gold (reference) executable and its usage documentation, a task worker is asked to write source code and a build script that constructs a candidate executable which should reproduce the behavior of the gold executable."

The test cases are built from an AI doing an examination of the source code and producing test cases, and later text also confirms that the AI during the production phase can't read the original executable so it can't reverse engineer it directly, so the test cases are being drawn from a situation where the tester has vastly more knowledge of the program than the implenter.

That is a losing scenario for anyone, be they human, modern AI, or even some hypothetical perfect programmer. Take ffmpeg as an extreme example. The documentation does not even remotely specify the program. Entire codecs can be missed at a stroke, and each of those codecs is itself a rich set of features that may or may not be used in a given input or output file, but the final tests can freely draw from any of those things. And trying to implement a codec from just some input and output would strain anyone, especially when the input is all but certain to not be sufficiently broad to make the determination for sure.

That sort of issue extends all the way down to even some tiny command-line programs I've written myself. The end-user documentation is never a specification. That's not what end-user documentation is. And even if you did hand the AI all relevant specifications you'd still get an implementation of the specification, but anyone who has ever implemented a non-trivial specification into real-world situations can tell you all about how even the spec is never enough.

I think that's an absolutely ridiculous test. If you handed to me as a human I would simply refuse because I'd tell you straight up front that it is plainly obvious I'm going to utterly and completely fail, so why even bother with the time to try?

Re: ProgramBench: Can language models rebuild programs from scratch?

#58

Post by tadamcz »

LeCompteSftware wrote: post #52
tadamcz wrote: post #44

Earlier quoted context omitted.

> Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. This is incidental to the main disagreement, but btw I also doubt this. Let's try and make the claim more precise. e.g. are you saying the average university undergraduate studying CS would reimplement cal from scratch (only stdlib), matching the output perfectly for all 1365 MirrorCode test cases, in (say) 3 days of…

I didn't say "3 days of full-time work," that is totally unreasonable. I was giving them basically unlimited time to do whatever slow testing and research they needed. And let me qualify my statement: when I say "I would expect most sophomores to be able to do this," I mean "if most sophomores can't do this then their university is badly failing them." (If you want to split hairs about modern undergrads not learning…

I think we're talking past each other here...

Re: ProgramBench: Can language models rebuild programs from scratch?

#59

Post by tadamcz »

stingraycharles wrote: post #37
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

Problem with these types of benchmarks is that it’s 100% certain the LLM has been trained on all that code already, so they’re all tainted since you don’t know whether it’s just benchmarking recall vs actual reasoning. Same with SWE-bench and others.

I agree it's a potentially big problem, affecting almost any benchmark out there. We discuss it briefly in "Appendix A: Contamination and memorization" https://epoch.ai/blog/mirrorcode-preliminary-results#appendi....

Ideally one would do these benchmarks with held-out proprietary software, but that comes with many practical concerns.

Re: ProgramBench: Can language models rebuild programs from scratch?

#60

Post by r123anH »

killerstorm wrote: post #53

It's a very misleading: they don't provide any meaningful documentation/requirements. Just an executable blackbox. E.g. a doc for ffmpeg, which I checked by downloading docker image they provide to the model, is a README which basically just says this is ffmpeg and docs can be found online. They do not allow models to get online. So a model is supposed to reverse-engineer a blackbox using only limited number of tries…

Ahem. Bitkeeper and Samba were reverse engineered just from the protocol by humans. For free.
Post reply on HN