Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ProgramBench: Can language models rebuild programs from scratch?

arxiv.org

Post reply on HN
Print view View original

21–30 of 86 posts

Re: ProgramBench: Can language models rebuild programs from scratch?

#21

Post by doix »

_pdp_ wrote: post #5

I am not surprised but this one sticks out... > Models favor monolithic, single-file implementations that diverge sharply from human-written code. Well, all of our code is monolithic with some files close 20K lines of code and we do use coding agents - not for the original code but as of late. I've always had that hunch that splitting everything into tiny files does not improve AI coding agent performance although it…

> Scattering the implementation in various files all over the source tree does not help much building the mental model.

Yeah, that happens where I work and I hate it. A combination of lint rules and AI reviewer prompts complain about long files and long functions. This means something that could be a 300 line self contained function that could be read linearly, gets split up into 6 functions across 6 files.

It's the illusion of "clean code". If you're casually skimming the code, you feel good. But as soon as you go beyond the surface level it becomes annoying.

Re: ProgramBench: Can language models rebuild programs from scratch?

#22

Post by aforwardslash »

osti wrote: post #10
quinnjh wrote: post #7

Earlier quoted context omitted.

My hunch is that it would take years of hundreds of thousands of developers working with machine code, posting stackoverflow questions with machine code, and publishing github repos written on it with documentation. Thats all the free labor LLMs leveraged to use high level langs. >We won't be developers, we won't be devops, we'll be modelops! /s I can still see this happening with higher level langs. the thing is the…

I heard they are already proficient at assembly languages.

They are - probably more proficient than with some high-level languages. I've used it for embedded stuff, including TI sitara PRU assembly, with great results. Frontier models can also easily "learn" directly from the manuals; asm is quite easy for them to pick up due to its "flat" (non-structured) nature.

Re: ProgramBench: Can language models rebuild programs from scratch?

#24

Post by tnelsond4 »

BurningPenguin wrote: post #11
_pdp_ wrote: post #5

I am not surprised but this one sticks out... > Models favor monolithic, single-file implementations that diverge sharply from human-written code. Well, all of our code is monolithic with some files close 20K lines of code and we do use coding agents - not for the original code but as of late. I've always had that hunch that splitting everything into tiny files does not improve AI coding agent performance although it…

Kinda surprising to me, since i had some trouble with Cursor & Co. once the file went over ~800 lines. It repeatedly failed to edit it, until i split it up into multiple logical components. As it should have been from the beginning... Though, it was some time ago, so things might have improved?

Yeah, that was my experience with Grok, whenever I gave it a file with over 400 lines it would just fail to comprehend it or be too lazy to write too much at a time. Splitting stuff up into separate files helped.

Re: ProgramBench: Can language models rebuild programs from scratch?

#25

Post by librasteve »

_pdp_ wrote: post #5

I am not surprised but this one sticks out... > Models favor monolithic, single-file implementations that diverge sharply from human-written code. Well, all of our code is monolithic with some files close 20K lines of code and we do use coding agents - not for the original code but as of late. I've always had that hunch that splitting everything into tiny files does not improve AI coding agent performance although it…

this is a big frustration for web code what with HTML, CSS, JS, PHP all spread about

https://htmx.org/essays/locality-of-behaviour/ is a good fight back as exemplified in many stacks, eg https://harcstack.org

Re: ProgramBench: Can language models rebuild programs from scratch?

#26

Post by logicchains »

_pdp_ wrote: post #5

I am not surprised but this one sticks out... > Models favor monolithic, single-file implementations that diverge sharply from human-written code. Well, all of our code is monolithic with some files close 20K lines of code and we do use coding agents - not for the original code but as of late. I've always had that hunch that splitting everything into tiny files does not improve AI coding agent performance although it…

> Models favor monolithic, single-file implementations that diverge sharply from human-written code.

This isn't the case if models are prompted to actually plan the file architecture beforehand, it's only the case if they're given a dumb monolithic "code this thing" prompt.

Re: ProgramBench: Can language models rebuild programs from scratch?

#27

Post by behaviors »

It's funny, because that task is very diverse. Any LLM will use the codebase given as a template(At least in free-tier models)

My software as a contract of behaviors works like a program bench(I even cross tested buildouts) Made an entire corpus layout for multi agent multi platform builds to be compared. Even went ahead and ran 50 contracts for an example. It honestly showed improvable areas, and distinct differences between model code.

{contract_name}/ └── submissions/ └── {date}_{os}_{agent}_{model}_{stack}/ ├── {contract}.osc.md ├── osc.osc.md └── results/ └── {contract}.snapshot.json That's it, compare to the same contract, or find a new contract to use to compare. Lot's of signed/hash pinned files are all you need to reproduce software from nothing, with an LLM.

Programbench is close to that(they have a nice paper/article here. But I don't like the work used. Having software to start with is not a bench of making code but reverse engineering.

github/s1ugh34d/osc

Re: ProgramBench: Can language models rebuild programs from scratch?

#28

Post by vrganj »

keyle wrote: post #4

How long until AI is not even writing code but producing machine code? Think about it, all these compilers, tooling, what a waste! I imagine a future where chipset makers will provide a model you can just prompt to "act upon that chipset" and voila, "You're absolutely right! Here is your binary." We won't be developers, we won't be devops, we'll be rollmops! /s

Good luck reasoning about the output in any meaningful way then. AI introduces a bug? Well, you're fucked.

Re: ProgramBench: Can language models rebuild programs from scratch?

#29

Post by tadamcz »

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air.

> Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task

Fwiw, this is very different from what we find in MirrorCode:

> Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benchmark.

https://epoch.ai/blog/mirrorcode-preliminary-results

I don't have time right now to dig in to what could explain the difference (I'm working hard on getting the full MirrorCode out as soon as possible). But I suspect that the ProgramBench authors are either under-eliciting the AIs, or their tasks are unfair/impossible given the constraints, or both.

I hope to look more into it after releasing MirrorCode, and write up my conclusions.

Re: ProgramBench: Can language models rebuild programs from scratch?

#30

Post by qsera »

aforwardslash wrote: post #22
osti wrote: post #10

Earlier quoted context omitted.

I heard they are already proficient at assembly languages.

They are - probably more proficient than with some high-level languages. I've used it for embedded stuff, including TI sitara PRU assembly, with great results. Frontier models can also easily "learn" directly from the manuals; asm is quite easy for them to pick up due to its "flat" (non-structured) nature.

>Frontier models can also easily "learn" directly from the manuals;

Really? So you just include the manual in the context? Or how does that work?

Post reply on HN