HarnessTax: How Much Does the Harness Matter for Coding Agents?

Post by faangguyindia » Thu, Sep 17, 2026, 10:41 AM UTC

ygouzerh wrote: post #42

corv wrote: post #11

My own findings are in line with this research: Having a coding harness is critical but the differences between them are overstated. Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes They’d all do the job - I just prefer to compartmentalize for access control. Keeping the harness’ surface area tiny had the added benefit of preser…

One thing I am missing to be able to move out of Claude Code, is the auto mode (and the soft_deny and hard_deny settings that can be tuned), with it's classifier checking the output. It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod. Other harnesses like Codex have o…

Post by ebrahimisoheil » Thu, Sep 17, 2026, 11:22 AM UTC

pyfd wrote: post #37

The part of my harness that earns its tokens isn't instructions, it's the bits that take claims out of the model's hands. Two examples from daily use on a small business's systems. Telling it to batch browser calls in the system prompt didn't hold; a hook that denies the seventh consecutive single call did. Telling it to record which machine did a piece of work didn't hold either; having the CLI stamp the hostname di…

Post by ebrahimisoheil » Thu, Sep 17, 2026, 11:24 AM UTC

joshheitzman wrote: post #13

ed_mercer wrote: post #2

Does this extend to open models like GLM 5.3? This would mean that simply changing the harness to Pi reduces cost in half?

The provider's middleware also plays a role. I just completed some benchmarks on my bespoke harness and Kilo Code. There's a chart on my LI post here: https://www.linkedin.com/posts/joshheitzman_i-ran-a-fuller-r... In the case of DeepSeek-V4-Flash-0731 on deepinfra.com there was little difference when both used high reasoning. In the case of that same model on together.ai there was a substantial difference between th…

Post by tomrod » Thu, Sep 17, 2026, 12:34 PM UTC

nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

Post by edgyquant » Thu, Sep 17, 2026, 12:38 PM UTC

mikert89 wrote: post #7

As the model gets smarter, you need to tell it less