Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HarnessTax: How Much Does the Harness Matter for Coding Agents?

harnesstax.github.io

Post reply on HN
Print view View original

41–50 of 76 posts

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#41

Post by skirmish »

zackify wrote: post #18
roywiggins wrote: post #12

Earlier quoted context omitted.

It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens. (I do think Pi should ship with more than zero builtin sandboxing though)

I have a couple hundred line TUI that just clones an LXD container and I get all my repos and infra isolated in 5s with pi having isolation to that container. Swap between any instance. Port forward to host all automatic. Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated a…

I tried putting my agent in an LXC / Incus container but then I also needed it to process videos stored on a NAS, and that was not easy at all. I tried a FUSE-based NFS client [1] but it was very flaky and would randomly report files having zero length and fail processing.

[1] https://github.com/facebookarchive/nfusr

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#42

Post by ygouzerh »

corv wrote: post #11

My own findings are in line with this research: Having a coding harness is critical but the differences between them are overstated. Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes They’d all do the job - I just prefer to compartmentalize for access control. Keeping the harness’ surface area tiny had the added benefit of preser…

One thing I am missing to be able to move out of Claude Code, is the auto mode (and the soft_deny and hard_deny settings that can be tuned), with it's classifier checking the output.

It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod.

Other harnesses like Codex have often on static rules, like the allow/deny of claude code, that can filter out based on regex. It's quite good already, but sometimes the model can find a way to write something that wasn't anticipated, or in a convoluted way.

After, I guess it's something that can be added in an open-source harness like Pi, and add like this new Jev model or something else equivalent

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#43

Post by vintagedave »

> Models may perform better with other harnesses than with their own. So it turns out that your Claude models may not need Claude Code…

This is interesting. We built our own harness (CodeBot, an agent for Delphi) and it currently uses OpenAI models; we tuned it for specific behaviours and patterns and I find its behaviour better than Codex. Same models underneath.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#44

Post by sandeepkd »

jswelker wrote: post #3

Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality. "Why pay the waste disposal tax? Dumping into the ocean is free!" Pi actively omits any sort of guardrails and sandboxing in the name o…

Wonder if you have any source to back that data, unless you are referring to the security code to detect the user and their usage itself to prevent against unauthorized usage patterns?

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#45

Post by sandeepkd »

nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

Harness and benchmark for the harness feels like a chicken and egg problem. The harness is to optimize the interaction results with the models. Any benchmark for harness has to focus on the goals that the harness was trying to optimize for unless we are only focussing on generic harnesses.

At this point when all the models have been trained on all available data with the similar algorithm,

1. either you get more data which is not feasible,

2. or get a better algorithm - a possibility ,

3. or write a more targeted harness.

Harnesses for legal, medicine and all are the ones which are getting focus for this reason. Writing benchmarks for these targeted harnesses would be a catching task

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#46

Post by corv »

ygouzerh wrote: post #42
corv wrote: post #11

My own findings are in line with this research: Having a coding harness is critical but the differences between them are overstated. Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes They’d all do the job - I just prefer to compartmentalize for access control. Keeping the harness’ surface area tiny had the added benefit of preser…

One thing I am missing to be able to move out of Claude Code, is the auto mode (and the soft_deny and hard_deny settings that can be tuned), with it's classifier checking the output. It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod. Other harnesses like Codex have o…

I believe Hermes offers something comparable now.

Personally, I’m not convinced that a cascade of LLM judges lets me sleep soundly at night.

My agents stay in their sandbox and escalating changes to that env or prod is solely in the (human) operator’s domain.

Obviously this is less convenient but I think it’s a necessary tradeoff for security and correctness.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#48

Post by imtringued »

lukax wrote: post #30

What matters more is that you use the tools that the target model was fine-tuned on. E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar). It appears newer models are better at narive harness tool calls and worse at custom tools that look simila…

This is correct. People seem to get the wrong idea about why agentic coding is even a thing in 2026. The naive AI techno optimist which has basically displaced the vast majority of opinions on HN, thinks that the models got "smarter" [0]. No, the training distribution shifted towards training on agentic sessions which made certain forms of agentic coding "in-distribution".

We are still witnessing the same underlying problems of transformers.

[0] Think back to all the publicity stunts like the Hugging Face. They are meant to convince you that the agents have somehow progressed past the transformer limitations when those publicity stunts are actually expressions of transformer limitations.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#49

Post by kouteiheika »

jswelker wrote: post #3

Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality. "Why pay the waste disposal tax? Dumping into the ocean is free!" Pi actively omits any sort of guardrails and sandboxing in the name o…

> Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes.

Um, no? Have you actually read Claude Code's system prompt and tools?

Here's Fable's system prompt: https://github.com/navanchauhan/agent-autopsy/blob/6d9c00e54...

There are a few lines of what I would call "for security and alignment purposes", but it's not the majority. And the majority of the bloat is from the tool descriptions.

The system prompt in this case has pretty much nothing to do with the built-in guard rails, that is: you cannot sidestep Claude's guard rails by editing/replacing the system prompt (which you officially can with Claude Code's `--system-prompt-file`).

Post reply on HN