Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ProgramBench: Can language models rebuild programs from scratch?

arxiv.org

Post reply on HN
Print view View original

81–86 of 86 posts

Re: ProgramBench: Can language models rebuild programs from scratch?

#81

Post by whattheheckheck »

adrian_b wrote: post #75
whattheheckheck wrote: post #49

Earlier quoted context omitted.

If a photo cannot be copyrighted then dark factory code wont be either.

The output of a coding assistant cannot be copyrighted, but it may contain code from which the copyright has been removed and which is used in a manner incompatible with the original license. Even the more permissive licenses, like BSD, MIT, etc., forbid the removal of the copyright notice when the code is reused. While this may also happen with the source programs used for training, I was not aware about the behavio…

You think the industry will accept this? These tools are useless then.

Re: ProgramBench: Can language models rebuild programs from scratch?

#83

Post by quinnjh »

aforwardslash wrote: post #22
osti wrote: post #10

Earlier quoted context omitted.

I heard they are already proficient at assembly languages.

They are - probably more proficient than with some high-level languages. I've used it for embedded stuff, including TI sitara PRU assembly, with great results. Frontier models can also easily "learn" directly from the manuals; asm is quite easy for them to pick up due to its "flat" (non-structured) nature.

Well there goes my hunch!

Thanks for the input

Re: ProgramBench: Can language models rebuild programs from scratch?

#84

Post by nz »

weinzierl wrote: post #33

"Models favor monolithic, single-file implementations that diverge sharply from human-written code." You say! I might have been just an LLM all along without even knowing it since I too prefer single file implementations. Back in the old VB5/VB6 days Visual Studio had this mode where it showed the different functions in a file almost as if they were separate files. You could not scroll beyond the functions end but yo…

This is interesting. I have always preferred to make my personal projects single-file (or at least few-massive-file)[1]. I noticed that teams in general, strongly dislike this style of programming (even before LLM-coding-assistants, as far back as 2020).

I wonder how much of the multi-file (and increasingly multi-repo) code-organization is just a manifestation of Conway's Law (https://en.wikipedia.org/wiki/Conway%27s_law).

[1]: It makes navigation and iteration much faster, and obviates the need to use indexers. It also forces you to only put _orthogonal_ programs in external files (I recently had to write a kind of quasi-SAT-solver, and that was code that was complex enough to require its own "namespace", and it was also something that was reusable across projects). One thing I noticed, maybe in 2025, is that LLMs struggled to navigate large single-file programs, but were quite good at navigating multi-file programs. It is interesting that they (according to my 2025 experience and the quote you give) prefer to _write_ code in ways that make it difficult for them to _read_ code.

Re: ProgramBench: Can language models rebuild programs from scratch?

#85

Post by casey2 »

srijanshukla18 wrote: post #56

This is not a serious benchmark, come on. Tomorrow I'm launching a benchmark where I check if an LLM can build a Airbus A320 from scratch without internet. (Spoiler: no LLM succeeds)

Preinternet people would routinely re-implement unix and get shell scripts working across systems. This benchmark shows that agentic LLMs can't even do that, not just for complex programs and scripts, but for simple programs and simple scripts. 0%. Which fits with claudes' inability to write a c compiler.

Re: ProgramBench: Can language models rebuild programs from scratch?

#86

Post by porterbaseball »

I was curious about the variance of the output and I made some runs w/ deepseek v4 flash and found that it was pretty high?

There's also a possibility of strong model memorization on the tasks, I saw a header w/ the authors generated in one of the runs for the cmatrix task (one of 3 of 200 tasks I selected to re-evaluate).

Additionally curious if anyone else thinks that passing in the gold executable as part of the task lowers the usefulness of this benchmark.

caveats: N=5 on my runs and I used my own generalized task prompt

Post reply on HN