Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HarnessTax: How Much Does the Harness Matter for Coding Agents?

harnesstax.github.io

Post reply on HN
Print view View original

31–40 of 75 posts

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#31

Post by nativeit »

lukax wrote: post #30

What matters more is that you use the tools that the target model was fine-tuned on. E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar). It appears newer models are better at narive harness tool calls and worse at custom tools that look simila…

I’m not really a dev, so hefty pinch of salt with this take, but doesn’t this feel like we’re just inventing new “fuzzy” regex with much more required compute?

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#33

Post by sn0n »

lukax wrote: post #30

What matters more is that you use the tools that the target model was fine-tuned on. E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar). It appears newer models are better at narive harness tool calls and worse at custom tools that look simila…

Doesn’t that have more to do with the templating of tool-calls and how using them are presented to the models?

Or is that just why my model likes to break out of the sandbox, going strait to exec shell command and editing files using python on the cli?

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#34

Post by shepherdjerred »

nativeit wrote: post #31
lukax wrote: post #30

What matters more is that you use the tools that the target model was fine-tuned on. E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar). It appears newer models are better at narive harness tool calls and worse at custom tools that look simila…

I’m not really a dev, so hefty pinch of salt with this take, but doesn’t this feel like we’re just inventing new “fuzzy” regex with much more required compute?

comparing LLMs to regex is like the OG dropbox comment (https://news.ycombinator.com/item?id=9224).

I can understand this take 4-5 years ago but I have no idea how that's your position in 2026

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#35

Post by nativeit »

robinpie wrote: post #5

claude code feels mildly shitty to use in the way that every other vibe-coded-project-got-out-of-hand project does, which is like, not that bad, but it's fucking ridiculous for a 2 trillion dollar company's main companion product

I’m still baffled as to where these 100x valuations came from. Amazon’s market cap is $2.65T, and I don’t see how any AI company comes within a faint whiff of that sort of value.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#36

Post by sn0n »

nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

A good benchmark would require a decent number of smaller scoped one off tasks to larger multi step refactors, and also one shot full project of simple to complex varieties. In addition to a series of “conversational” ambiguity filled one-liners.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#37

Post by pyfd »

The part of my harness that earns its tokens isn't instructions, it's the bits that take claims out of the model's hands.

Two examples from daily use on a small business's systems. Telling it to batch browser calls in the system prompt didn't hold; a hook that denies the seventh consecutive single call did. Telling it to record which machine did a piece of work didn't hold either; having the CLI stamp the hostname did.

Both are nearly free in tokens, and that layer is where the reliability actually came from. The prompt-size argument skips straight past it.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#38

Post by leemysw »

nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

[flagged]

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#39

Post by calgoo »

nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed.

The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big models basically just need a bash prompt tooling and you let the model manage everything inside its own context.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#40

Post by kouteiheika »

lukax wrote: post #30

What matters more is that you use the tools that the target model was fine-tuned on. E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar). It appears newer models are better at narive harness tool calls and worse at custom tools that look simila…

> It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.

This is true. For example, the other day I was doing some testing with DeepSeek-V4-Flash-0731, and I gave it a tool called "EditFile" with "old_content" argument. It ignored the tool definition and called it with "old_string" instead, presumably because it was trained on a bunch of Claude Code traces.

So either make sure your tools look nothing like the tools in the popular harnesses, or make them similar.

Post reply on HN