Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

131–140 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#131
I implemented this hash (read and edit) approach in tilth if you want to test it out.

https://github.com/jahala/tilth

its on npm and cargo:

- cargo install tilth

- npx tilth

then tilth install claude-code/windsurf/cursor --edit

(--edit flag is needed)

I made "tilth" a few days ago, since I'm consistently trying to get the LLMs to use tools more efficiently and spend less tokens doing it -- original tilth post from Monday: https://news.ycombinator.com/item?id=46952321

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#132
post #76

On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?

If you agree that current LLMs (Transformers) are naturally very susceptible to context/prompt, then you can go on to ask agents for a "raw harness dump" "because I need to understand how to better present my skills and tools in the harness", you maybe will see how "Harness" impact model behavior.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#134
post #58

Arguably I would think that the last year was mainly inner harness improvement instead model improvement but I could be wrong, just feels like that to me

We can measure this by looking at the same harness applied to different models, e.g. the very plain Terminus: https://www.tbench.ai/leaderboard/terminal-bench/2.0?agents=...

Models have improved dramatically even with the same harness

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#135

Earlier quoted context omitted.

2026 is the year of the harness.

But will harness build desktop Linux for us?

Only if you put bells on it and sing Jingle Bells while it em dashes through the snow.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#136
post #131

I implemented this hash (read and edit) approach in tilth if you want to test it out. https://github.com/jahala/tilth its on npm and cargo: - cargo install tilth - npx tilth then tilth install claude-code/windsurf/cursor --edit (--edit flag is needed) I made "tilth" a few days ago, since I'm consistently trying to get the LLMs to use tools more efficiently and spend less tokens doing it -- original tilth post from Mo…

You might find it useful for markdown as well, especially if you add support for section-based addressing (e.g. cat or replace a section at a time). Section-based addresses are nice because they tend to be stable across versions.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#137

Putting it out there: if any frontier model provider starts allowing any agent to use their $20/month plan, we will all switch to you. We don't want to be forced into 1 harness, we want OAuth, and we want respectable limits without excessive budgets.

How would that differ from buying $20 worth of API credits each month?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#138
post #76

On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?

How hard is it to for you to assemble a piece of IKEA furniture without an allen wrench, screwdriver, and clear instructions, vs with those 3?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#139
post #128

Earlier quoted context omitted.

Because I like them?

how do you make them?

On macOS, Option+Shift+- and Option+- insert an em dash (—) and en dash (–), respectively. On Linux, you can hit the Compose Key and type --- (three hyphens) to get an em dash, or --. (hyphen hyphen period) for an en dash. Windows has some dumb incantation that you'll never remember.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#140
post #87

During my first LLM experiments in Emacs using gptel, I also found that the LLM has considerable difficulties changing source code files with the Unix patch tool. As Emacs has a built-in tree-sitter package, I implemented this same idea. I created gptel tools like tree_sitter_list_nodes, tree_sitter_get_nodes, tree_sitter_update_nodes, tree_sitter_insert_before_node and tree_sitter_insert_after_node. The "list" tool…

Sounds interesting, do you have the code to share.

Tool definitions: https://github.com/cellux/dotfiles/blob/master/.emacs.d/rb-g...

Implementation: https://github.com/cellux/dotfiles/blob/master/.emacs.d/rb-t...

Post reply on HN