[dead]
HarnessTax: How Much Does the Harness Matter for Coding Agents?
21–30 of 75 posts
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#22Say I'm using Claude Code or GPT Codex's harnesses but also sending some queries to the respective Anthropic and OpenAI models via OpenRouter. Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if th…
Your question is worded kind of confusingly, but all caching is handled on the inference layer, and by all major providers. In short, caching should work as long as you are sending requests to the same model and provider.
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#23We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.
I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?
My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#24Earlier quoted context omitted.
It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens. (I do think Pi should ship with more than zero builtin sandboxing though)
I have a couple hundred line TUI that just clones an LXD container and I get all my repos and infra isolated in 5s with pi having isolation to that container. Swap between any instance. Port forward to host all automatic. Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated a…
Do you have any links that would explain this?
I'm looking at changing my AI harness and am definitely interested in this
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#25[dead]
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#26Say I'm using Claude Code or GPT Codex's harnesses but also sending some queries to the respective Anthropic and OpenAI models via OpenRouter. Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if th…
Your question is worded kind of confusingly, but all caching is handled on the inference layer, and by all major providers. In short, caching should work as long as you are sending requests to the same model and provider.
Apologies. I guess my main question is just is there a benefit to routing directly to a provider through its own harness vs just sending chat requests to those providers using OpenRouter or similar as a proxy?
As per your response it sounds like at least caching would happen for any provider regardless of the request's origin.
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#27I think it’s really important, especially when you look at everything the tool does, from the execution loop and context management to feedback. The harness is basically the underlying source of truth.
With coding agents, what I’ve noticed is that a simple task can often be handled with a fairly simple harness. But the hidden cost is really around context. One of the more interesting things I’ve seen is that two different harnesses can make a similar number of model calls while consuming a very different amount of context.
I think I recently came across a paper comparing Claude Code and Pi that touched on this. More context, more tooling, focused context, simpler loops, all of these can lead to very different costs and performance, even when the number of model calls looks similar.
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#28I have been using jcode for past two weeks, honestly I feel its much better compared to Opencode.
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#29The term "harness" here is being overloaded for the term "agent", which is worrying. Putting that aside, there are many factors that matter. The "harness" context, the execution pattern (parallel vs sequential), the ability to delegate to other models, etc. Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of t…
While sub-agents could theoretically stack turtles indefinitely, there’s limited room when going up the stack because someone will have to take responsibility beyond “git blame”, and that lands squarely with a person or legal entity.
Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?
#30What matters more is that you use the tools that the target model was fine-tuned on.
E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar).
It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.
https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/