Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HarnessTax: How Much Does the Harness Matter for Coding Agents?

harnesstax.github.io

Post reply on HN
Print view View original

51–60 of 83 posts

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#51

Post by general_reveal »

nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

Any harness will always be privately and secretly shaped and created by those selling models, especially coding models. It is literally a “selling point”, and unless the government steps in to oversee the tests like in the car industry, then there is absolutely no way lizard satanists like Altman and Musk are going to exercise their native ethical traits (“native” loosely assumes they procured it divinely and quite recently, because, we simply haven’t observed it prior). Short of that, these tests will be fabricated, a lot, for money.

Tell a horny monkey not to jerk off. How the fuck … would that even be possible? God, only God can stop this godless train.

There is no sincere discussion to be had here. HN has been a cesspool for marketing and it reeks in here lately.

Edit: I am not punching down, it’s gotta be crooks from those companies down-voting.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#52

Post by raffael_de »

kouteiheika wrote: post #40
lukax wrote: post #30

What matters more is that you use the tools that the target model was fine-tuned on. E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar). It appears newer models are better at narive harness tool calls and worse at custom tools that look simila…

> It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools. This is true. For example, the other day I was doing some testing with DeepSeek-V4-Flash-0731, and I gave it a tool called "EditFile" with "old_content" argument. It ignored the tool definition and called it with "old_string" instead, presumably because it was trained on a bunch of Claude Co…

> or make them similar.

identical. similar is what you tried and it didn't work.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#53

Post by raffael_de »

nativeit wrote: post #31
lukax wrote: post #30

What matters more is that you use the tools that the target model was fine-tuned on. E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar). It appears newer models are better at narive harness tool calls and worse at custom tools that look simila…

I’m not really a dev, so hefty pinch of salt with this take, but doesn’t this feel like we’re just inventing new “fuzzy” regex with much more required compute?

> inventing new “fuzzy” regex

not RegEx ... more like a "fuzzy" DSL.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#54

Post by TedDoesntTalk »

imtringued wrote: post #48
lukax wrote: post #30

What matters more is that you use the tools that the target model was fine-tuned on. E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar). It appears newer models are better at narive harness tool calls and worse at custom tools that look simila…

This is correct. People seem to get the wrong idea about why agentic coding is even a thing in 2026. The naive AI techno optimist which has basically displaced the vast majority of opinions on HN, thinks that the models got "smarter" [0]. No, the training distribution shifted towards training on agentic sessions which made certain forms of agentic coding "in-distribution". We are still witnessing the same underlying…

You think the hugging face incident was a stunt? Can you explain?

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#55

Post by imtringued »

roywiggins wrote: post #12
jswelker wrote: post #3

Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality. "Why pay the waste disposal tax? Dumping into the ocean is free!" Pi actively omits any sort of guardrails and sandboxing in the name o…

It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens. (I do think Pi should ship with more than zero builtin sandboxing though)

The developers of pi.dev are geniuses.

They add a --tools flag, which can only add tools onto the built-in tools, so if you wanted to sandbox pi.dev by adding sandboxed versions of the existing tools you will have to run pi.dev via

    pi --no-tools --tools tool1, tool2, tool3
except..., that's not enough to sandbox pi.dev, because remember, it's a minimalist coding agent! So what does a minimalist coding agent do? Of course! It loads extensions by default and do you know what extensions can do? they can add default tools that bypass --no-tools!

So if you want to sandbox your agent, guess what you'll have to do? Yep, you have to supply --no-extensions.

Here is how to run pi.dev under its most minimal configuration under a sandbox:

    pi --no-tools --no-extensions --tools tool1, tool2, tool3 -e ./your-sandboxing-extension
Pretty neat, huh?

Sources: https://github.com/earendil-works/pi/issues/555

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#56

Post by kouteiheika »

raffael_de wrote: post #52
kouteiheika wrote: post #40

Earlier quoted context omitted.

> It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools. This is true. For example, the other day I was doing some testing with DeepSeek-V4-Flash-0731, and I gave it a tool called "EditFile" with "old_content" argument. It ignored the tool definition and called it with "old_string" instead, presumably because it was trained on a bunch of Claude Co…

> or make them similar. identical. similar is what you tried and it didn't work.

No. You don't need identical. For example, I have extra arguments in my edit file tool that the model can specify, and that works just fine.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#57

Post by mobelkh »

calgoo wrote: post #39
nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed. The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big…

what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#58

Post by visarga »

corv wrote: post #29
Supermancho wrote: post #4

The term "harness" here is being overloaded for the term "agent", which is worrying. Putting that aside, there are many factors that matter. The "harness" context, the execution pattern (parallel vs sequential), the ability to delegate to other models, etc. Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of t…

While sub-agents could theoretically stack turtles indefinitely, there’s limited room when going up the stack because someone will have to take responsibility beyond “git blame”, and that lands squarely with a person or legal entity.

A big part of that responsibility can be put in code tests.

Ensuring good test coverage and quality is how you purchase trust in the work agents do. This also reduces the context problem - a test collection has no recall issues, it just runs every time you call it, the whole battery, checks all the things we could check by code in one fast tool call. For the rest, the things we can't test by code, I use manual testing.

A large class of problems are intent divergence, when the model passes tests but it didn't do what I asked. For that I keep a log of all user messages in the project history and review it with agents. This intent alignment is repeated from time to time to catch drift.

So I see the "why should I trust the work agent did?" problem as a combination of 1. ensure good testing 2. review intent alignment.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#59

Post by anon373839 »

corv wrote: post #11

My own findings are in line with this research: Having a coding harness is critical but the differences between them are overstated. Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes They’d all do the job - I just prefer to compartmentalize for access control. Keeping the harness’ surface area tiny had the added benefit of preser…

> Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes

That's really interesting. I like Pydantic AI a lot and wondered why all of the harnesses seem to be written in Javascript instead of it. What do you use it for headless, though? I haven't tried Hermes or similar yet, so don't have a handle on what you do with them.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#60

Post by tacomagick »

mobelkh wrote: post #57
calgoo wrote: post #39

Earlier quoted context omitted.

I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed. The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big…

what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?

Moving from CC I can agree OpenCode seems very simple. I don't see what is too much about it honestly.
Post reply on HN