Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ProgramBench: Can language models rebuild programs from scratch?

arxiv.org

Post reply on HN
Print view View original

41–50 of 86 posts

Re: ProgramBench: Can language models rebuild programs from scratch?

#41

Post by grego »

tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

Is anyone familiar with gotree? That was mentioned as the most complex piece of code, but the metric was LOC. Based on the high level description gotree might be closer to a set of small programs / algorithms.

Interesting anyway. It will be nice to see these comparisons with open weight models and how do those fare.

Re: ProgramBench: Can language models rebuild programs from scratch?

#42

Post by [deleted] »

LeCompteSftware wrote: post #35
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

Surely the biggest difference is that you guys are mostly testing LLMs on simpler utilities, mostly involving higher-level languages, whereas ProgramBench are all very complex C programs (and much older programs with much more comprehensive test cases). Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. In fact the only program you tested which actually has anywhere cl…

[deleted]

Re: ProgramBench: Can language models rebuild programs from scratch?

#43

Post by bmn__ »

weinzierl wrote: post #33

"Models favor monolithic, single-file implementations that diverge sharply from human-written code." You say! I might have been just an LLM all along without even knowing it since I too prefer single file implementations. Back in the old VB5/VB6 days Visual Studio had this mode where it showed the different functions in a file almost as if they were separate files. You could not scroll beyond the functions end but yo…

This VB feature existed to accommodate programmers coming from the DOS based QB IDE who were used to the one function per screen view there. To my sensibilities, it does not make much sense with the advent of high-resolution desktop environments.

Re: ProgramBench: Can language models rebuild programs from scratch?

#44

Post by tadamcz »

LeCompteSftware wrote: post #35
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

Surely the biggest difference is that you guys are mostly testing LLMs on simpler utilities, mostly involving higher-level languages, whereas ProgramBench are all very complex C programs (and much older programs with much more comprehensive test cases). Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal. In fact the only program you tested which actually has anywhere cl…

> Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal.

This is incidental to the main disagreement, but btw I also doubt this.

Let's try and make the claim more precise. e.g. are you saying the average university undergraduate studying CS would reimplement cal from scratch (only stdlib), matching the output perfectly for all 1365 MirrorCode test cases, in (say) 3 days of full-time work (without AI assistance obviously)? I'd bet against it!

Here is the manual for the cal that we use: https://media.githubusercontent.com/media/epoch-research/Mir...

You can also look at a full transcript of an LLM solving the task: https://epochai-public-eval-logs-manual.s3.amazonaws.com/eva...

The data is here: https://github.com/epoch-research/MirrorCode-data/

Re: ProgramBench: Can language models rebuild programs from scratch?

#45

Post by tadamcz »

grego wrote: post #41
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

Is anyone familiar with gotree? That was mentioned as the most complex piece of code, but the metric was LOC. Based on the high level description gotree might be closer to a set of small programs / algorithms. Interesting anyway. It will be nice to see these comparisons with open weight models and how do those fare.

There's a more detailed description in "Appendix B: Qualitative discussion of the gotree task"

https://epoch.ai/blog/mirrorcode-preliminary-results#appendi...

Re: ProgramBench: Can language models rebuild programs from scratch?

#46

Post by tadamcz »

tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

I should say one big difference is ProgramBench has 200 target programs while MirrorCode has about 30. We did many manual things to ensure task quality, that would have required huge resources to do at ProgramBench scale.

Re: ProgramBench: Can language models rebuild programs from scratch?

#47

Post by rullopat »

weinzierl wrote: post #33

"Models favor monolithic, single-file implementations that diverge sharply from human-written code." You say! I might have been just an LLM all along without even knowing it since I too prefer single file implementations. Back in the old VB5/VB6 days Visual Studio had this mode where it showed the different functions in a file almost as if they were separate files. You could not scroll beyond the functions end but yo…

I think it's one (but not the only) reason that makes LLMs work very well with Ruby on Rails

Re: ProgramBench: Can language models rebuild programs from scratch?

#48

Post by phpnode »

VladVladikoff wrote: post #36
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

I would love to try this out. I have a horrible legacy project that is written in angular by a really amateur developer, full of huge blocks of copy pasted code that has minor modifications in each block. I’ve tried before to get an LLM to rewrite it to something more sensible, but I have not succeeded, usually it just ends up breaking everything. Is there a guide or some system to follow? What’s the best way to acco…

I have an approach that can handle this if you're interested? My email is in my profile.

Re: ProgramBench: Can language models rebuild programs from scratch?

#49

Post by whattheheckheck »

adrian_b wrote: post #34

> Open internet with cheating detection => cheating is widespread, 20-36% of tasks are flagged for the stronger models, with source code lookup accounting for the majority of the violations. Therefore: > blocking internet access entirely is the appropriate default for ProgramBench The fact that your Anthropic coding assistant has a tendency to search on the Internet code to be inserted into your program may count for…

If a photo cannot be copyrighted then dark factory code wont be either.
Post reply on HN