Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

141–150 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#141

Post by parhamn »

znnajdla wrote: post #138
parhamn wrote: post #76

On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?

How hard is it to for you to assemble a piece of IKEA furniture without an allen wrench, screwdriver, and clear instructions, vs with those 3?

You didn't read the article it seems (or the analogy is a bad one). The differences are much more subtle than having a screwdriver or not.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#143

Post by znnajdla »

babkayaga wrote: post #133

Still weird to me that most people are not just giving an LLM an access to an editor, forcing it to write shell scripts to edit files. Shrug.

How do you give it access to an editor? It doesn't have a keyboard and mouse.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#144

Post by visarga »

senordevnyc wrote: post #114
robbomacrae wrote: post #63

Earlier quoted context omitted.

Indeed. The biggest waste might be the overuse of MCP for everything. Sure it makes the initial development easier but then for every connection you're using a hundred billion dollar parameter model to decide how to make the call when it's usually completely unnecessary and then prone to random errors. MCP is the hammer that can make literally everything look like a nail...

I see this ranting against MCP all the time, and I don't get it, maybe I'm missing something. I'm currently using an MCP in Cursor to give agents read-only access to my staging and prod databases, as well as BugSnag's MCP so it can look up errors that happen in those environments. It works great . What should I be using for this if not MCP?

Make a CLI tool for it, of course

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#145

Post by kachapopopow »

jahala wrote: post #131

I implemented this hash (read and edit) approach in tilth if you want to test it out. https://github.com/jahala/tilth its on npm and cargo: - cargo install tilth - npx tilth then tilth install claude-code/windsurf/cursor --edit (--edit flag is needed) I made "tilth" a few days ago, since I'm consistently trying to get the LLMs to use tools more efficiently and spend less tokens doing it -- original tilth post from Mo…

benchmarks vs grep?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#146

Post by matheist »

> Codex uses apply_patch: It takes a string as input, which is essentially an OpenAI-flavored diff, and instead of relying on a structured schema, the harness just expects this blob to follow a strict set of rules. Since OpenAI folks are without a doubt smart, I’m sure the token selection process is biased to fit this structure at the LLM gateway for the Codex variants of GPT, similar to how other constraints like JSON schemas or required tool calls work.

Codex does in fact use a schema for constrained sampling, it's here: https://github.com/openai/codex/blob/main/codex-rs/core/src/...

It still has to work to get an exact match, or at least I didn't read the code to see if there's any fuzzy matching used.

Note the two codex models were the only ones doing worse with the author's proposed format. The author found them doing better with replace than with apply patch, but since the author appears to be unaware that they use a schema for constrained sampling, I think a more realistic benchmark should enable constrained sampling for the apply test.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#147

Post by cevn »

esafak wrote: post #54
brendanmc6 wrote: post #53

Earlier quoted context omitted.

You’re absolutely right! This isn’t your average engineering advice— it’s like painting the reader a vivid tapestry of the author’s mind.

Please stop; I just can't any more! Yes, I'm absolutely right.

You're absolutely right about being absolutely right!

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#148

Post by visarga »

znnajdla wrote: post #143
babkayaga wrote: post #133

Still weird to me that most people are not just giving an LLM an access to an editor, forcing it to write shell scripts to edit files. Shrug.

How do you give it access to an editor? It doesn't have a keyboard and mouse.

I built a structural zoom tool, it would fit flat or tree like content into a 10K char budget. It can compress HTML, JSON, folders, zip files, logs, chat sessions, basically large files or collections of files. Moving around is done by range selection. The idea is to have the agent find its way iteratively to the target, while having the structure exposed. RAG would totally cut everything to pieces and put them in a hat. My approach is to follow the structure of a large content by a series of glimpses. Unfortunately I myself am not sure it is better to use this tool vs bash and python one off scripts.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#149

Post by rao-v »

I’d really like to see this optimized for the 50-120B parameter open source models that are local viable (gpt-oss-120b, qwen3-80b-3a etc.).

For them I think it would be optimal to provide a tag per function and trust the llm to rewrite the function. As the article notes full reproduction is generally more reliable than edited for short code.

The token and attention overhead from a per line hash I suspect limits this approach for smaller models

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#150

Post by TuxSH »

Aurornis wrote: post #44
horsawlarway wrote: post #14

Earlier quoted context omitted.

Also another place where having it change out from underneath you can drastically alter the quality of your work in unexpected ways. Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly. Even if the "limits" on them stay generous, the product will start shifting to prioritize things the user doesn't want. Tool recommendations are my imm…

> Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly. I don’t assume this at all. In fact, the opposite has been happening in my experience: I try multiple providers at the same time and the $20/month plans have only been getting better with the model improvements and changes. The current ChatGPT $20/month plan goes a very long way ev…

> The current ChatGPT $20/month plan goes a very long way

It sure does and Codex is great, but do you think they'll maintain the current prices after/if it eventually dominates Claude Code in terms of marketshare and mindshare?

Post reply on HN