Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HarnessTax: How Much Does the Harness Matter for Coding Agents?

harnesstax.github.io

Post reply on HN
Print view View original

11–20 of 76 posts

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#11

Post by corv »

My own findings are in line with this research:

Having a coding harness is critical but the differences between them are overstated.

Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes

They’d all do the job - I just prefer to compartmentalize for access control.

Keeping the harness’ surface area tiny had the added benefit of preserving my understanding and being able to adapt it to my preferred workflow effortlessly

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#12

Post by roywiggins »

jswelker wrote: post #3

Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality. "Why pay the waste disposal tax? Dumping into the ocean is free!" Pi actively omits any sort of guardrails and sandboxing in the name o…

It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens.

(I do think Pi should ship with more than zero builtin sandboxing though)

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#13

Post by joshheitzman »

ed_mercer wrote: post #2

Does this extend to open models like GLM 5.3? This would mean that simply changing the harness to Pi reduces cost in half?

The provider's middleware also plays a role. I just completed some benchmarks on my bespoke harness and Kilo Code. There's a chart on my LI post here: https://www.linkedin.com/posts/joshheitzman_i-ran-a-fuller-r...

In the case of DeepSeek-V4-Flash-0731 on deepinfra.com there was little difference when both used high reasoning. In the case of that same model on together.ai there was a substantial difference between the two (high reasoning for both again). When using together.ai with Kilo Code the LLM was having a lot of trouble making successful edits. In some cases that meant a lot tries at using the tools and in others it worked around by running scripts. Meanwhile it used the tools from my harness just fine. I've specifically tried to make my tools easy for all of the open weight LLMs to use correctly. That was inspired by getting some errors from Kilo Code at the beginning of the year telling me that the model was having trouble and I should use a smarter model.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#14

Post by x312 »

Claude Code/Codex charge the user for their extremely bloated one-size-fits-all system prompts (including safety instructions and other stuff users dont want).

In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#15

Post by spike021 »

Say I'm using Claude Code or GPT Codex's harnesses but also sending some queries to the respective Anthropic and OpenAI models via OpenRouter.

Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if the requests go straight from the harness to the provider's API?

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#16

Post by seanmcdirmid »

roywiggins wrote: post #12
jswelker wrote: post #3

Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality. "Why pay the waste disposal tax? Dumping into the ocean is free!" Pi actively omits any sort of guardrails and sandboxing in the name o…

It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens. (I do think Pi should ship with more than zero builtin sandboxing though)

You can sandbox with tool hooks, at least in antigravity, I think a lot of other harnesses support similar features. Sandboxing via the prompt is just dangerous and unreliable.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#18

Post by zackify »

roywiggins wrote: post #12
jswelker wrote: post #3

Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality. "Why pay the waste disposal tax? Dumping into the ocean is free!" Pi actively omits any sort of guardrails and sandboxing in the name o…

It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens. (I do think Pi should ship with more than zero builtin sandboxing though)

I have a couple hundred line TUI that just clones an LXD container and I get all my repos and infra isolated in 5s with pi having isolation to that container. Swap between any instance. Port forward to host all automatic.

Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated all in one tooling.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#20

Post by topaz0 »

robinpie wrote: post #5

claude code feels mildly shitty to use in the way that every other vibe-coded-project-got-out-of-hand project does, which is like, not that bad, but it's fucking ridiculous for a 2 trillion dollar company's main companion product

It's pretty great to not use, conversely.
Post reply on HN