Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ProgramBench: Can language models rebuild programs from scratch?

arxiv.org

Post reply on HN
Print view View original

71–80 of 86 posts

Re: ProgramBench: Can language models rebuild programs from scratch?

#71

Post by kibwen »

tmtvl wrote: post #66
weinzierl wrote: post #33

"Models favor monolithic, single-file implementations that diverge sharply from human-written code." You say! I might have been just an LLM all along without even knowing it since I too prefer single file implementations. Back in the old VB5/VB6 days Visual Studio had this mode where it showed the different functions in a file almost as if they were separate files. You could not scroll beyond the functions end but yo…

How often has there been a HN submission for a project 'in a single C header file'?

This has less to do with natural opinions regarding code organization and more to do with the fact that including, modularizing, and distributing C code has historically been a pain in the ass which is ameliorated by shoving everything into a single file.

Re: ProgramBench: Can language models rebuild programs from scratch?

#72

Post by astrange »

VladVladikoff wrote: post #36
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

I would love to try this out. I have a horrible legacy project that is written in angular by a really amateur developer, full of huge blocks of copy pasted code that has minor modifications in each block. I’ve tried before to get an LLM to rewrite it to something more sensible, but I have not succeeded, usually it just ends up breaking everything. Is there a guide or some system to follow? What’s the best way to acco…

I've been very successful so far using Sonnet 4.6 (1M) as the basic model in Claude Code, plus Codex and gemini-review plugins for second/third opinions. (The last one is somewhat busted and hardcoded old gemini versions, I should patch it up.)

I needed to use Opus 4.7 for one project because it used very recent APIs, and it certainly is smart but it's also very expensive.

Re: ProgramBench: Can language models rebuild programs from scratch?

#73

Post by weinzierl »

tmtvl wrote: post #66
weinzierl wrote: post #33

"Models favor monolithic, single-file implementations that diverge sharply from human-written code." You say! I might have been just an LLM all along without even knowing it since I too prefer single file implementations. Back in the old VB5/VB6 days Visual Studio had this mode where it showed the different functions in a file almost as if they were separate files. You could not scroll beyond the functions end but yo…

How often has there been a HN submission for a project 'in a single C header file'?

EDIT: Sorry, I missed the "header" part (and the irony).

At least once, here you go:

https://news.ycombinator.com/item?id=48053570

Ok, I just submitted it myself but I could not believe it never had been submitted before. It is from 1997 and was pretty popular for some time. I think it even was built into Google Picasa for sime time.

Re: ProgramBench: Can language models rebuild programs from scratch?

#74

Post by westurner »

tmtvl wrote: post #66
weinzierl wrote: post #33

"Models favor monolithic, single-file implementations that diverge sharply from human-written code." You say! I might have been just an LLM all along without even knowing it since I too prefer single file implementations. Back in the old VB5/VB6 days Visual Studio had this mode where it showed the different functions in a file almost as if they were separate files. You could not scroll beyond the functions end but yo…

How often has there been a HN submission for a project 'in a single C header file'?

/? single header https://hn.algolia.com/?q=single%20header

/? header-only https://hn.algolia.com/?q=header-only

/? c header https://hn.algolia.com/?q=c+header

Re: ProgramBench: Can language models rebuild programs from scratch?

#75

Post by adrian_b »

whattheheckheck wrote: post #49
adrian_b wrote: post #34

> Open internet with cheating detection => cheating is widespread, 20-36% of tasks are flagged for the stronger models, with source code lookup accounting for the majority of the violations. Therefore: > blocking internet access entirely is the appropriate default for ProgramBench The fact that your Anthropic coding assistant has a tendency to search on the Internet code to be inserted into your program may count for…

If a photo cannot be copyrighted then dark factory code wont be either.

The output of a coding assistant cannot be copyrighted, but it may contain code from which the copyright has been removed and which is used in a manner incompatible with the original license.

Even the more permissive licenses, like BSD, MIT, etc., forbid the removal of the copyright notice when the code is reused.

While this may also happen with the source programs used for training, I was not aware about the behavior described in TFA for the Anthropic agents, which may search the Internet for source code applicable to the problem that must be solved. It seems even more likely that such code will not be used as allowed by its license.

Re: ProgramBench: Can language models rebuild programs from scratch?

#76

Post by jaggederest »

VladVladikoff wrote: post #36
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

I would love to try this out. I have a horrible legacy project that is written in angular by a really amateur developer, full of huge blocks of copy pasted code that has minor modifications in each block. I’ve tried before to get an LLM to rewrite it to something more sensible, but I have not succeeded, usually it just ends up breaking everything. Is there a guide or some system to follow? What’s the best way to acco…

I think one way is to take the existing system in something like a docker container or equivalent, some kind of black box, and write tests against it in pure HTTP calls or using browser automation to record (can drive it with AI). When you've reached a truly massive test suite that covers everything, you delete the container and use the test suite as an oracle for writing a new version (open book, the AI can look at the test suite but not change it).

This is a tactic based on things I have read in "Working Effectively with Legacy Code" by Michael Feathers - he discusses using cut points to build a testing firewall to bring code under test, then gradually expanding the test suite from that beachhead of confirmed interface.

Re: ProgramBench: Can language models rebuild programs from scratch?

#77

Post by aforwardslash »

qsera wrote: post #30
aforwardslash wrote: post #22

Earlier quoted context omitted.

They are - probably more proficient than with some high-level languages. I've used it for embedded stuff, including TI sitara PRU assembly, with great results. Frontier models can also easily "learn" directly from the manuals; asm is quite easy for them to pick up due to its "flat" (non-structured) nature.

>Frontier models can also easily "learn" directly from the manuals; Really? So you just include the manual in the context? Or how does that work?

Yes, something like "analyze throroughly the @datasheet.pdf and create a plan to implement x"

Re: ProgramBench: Can language models rebuild programs from scratch?

#79

Post by stingraycharles »

Zigurd wrote: post #51
stingraycharles wrote: post #37

Earlier quoted context omitted.

Problem with these types of benchmarks is that it’s 100% certain the LLM has been trained on all that code already, so they’re all tainted since you don’t know whether it’s just benchmarking recall vs actual reasoning. Same with SWE-bench and others.

That's a feature not a bug. It doesn't make benchmarking any more meaningful or simple, but being trained to recall patterns is a legitimate goal for a coding agent.

Yes but then the benchmarks need to be presented as "this verifies whether the model can recall this exact same situation and does not actually benchmark any reasoning at all".

This is not the case, they're being presented as "how good is the model at software engineering". E.g. the benchmark in question says this:

"Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. "

When your benchmark is fundamentally embedded extremely well in the training data, such that you're actually just benchmarking "how well do you remember what sqlite looks like" rather than "do you understand all the tradeoffs, risks, design decisions that need to be made to build a bespoke database from scratch".

This is a VERY big caveat that, to me, for a decent part explains the discrepancy between benchmarks and reality.

Re: ProgramBench: Can language models rebuild programs from scratch?

#80

Post by rurban »

VladVladikoff wrote: post #36
tadamcz wrote: post #29

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air. > Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task Fwiw, this is very different from what we find in MirrorCode: > Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benc…

I would love to try this out. I have a horrible legacy project that is written in angular by a really amateur developer, full of huge blocks of copy pasted code that has minor modifications in each block. I’ve tried before to get an LLM to rewrite it to something more sensible, but I have not succeeded, usually it just ends up breaking everything. Is there a guide or some system to follow? What’s the best way to acco…

Normal engineering practices as thought since the 70ies.

Break the problems up into manageable pieces. Make a plan, have tests to verify the outcome, implement that part. Rinse and repeat. Have integration tests.

Post reply on HN