Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HarnessTax: How Much Does the Harness Matter for Coding Agents?

harnesstax.github.io

Post reply on HN
Print view View original

71–80 of 89 posts

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#71

Post by vhantz »

jswelker wrote: post #3

Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality. "Why pay the waste disposal tax? Dumping into the ocean is free!" Pi actively omits any sort of guardrails and sandboxing in the name o…

Claude Code's initial instructions to a model are a dump of 80kb of text. And none of it has anything to do with security. The "security" is handled by the evaluator model that sometimes denies some tool uses. I'm sure that in itself can be done very cheaply.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#72

Post by CharlieDigital »

robinpie wrote: post #5

claude code feels mildly shitty to use in the way that every other vibe-coded-project-got-out-of-hand project does, which is like, not that bad, but it's fucking ridiculous for a 2 trillion dollar company's main companion product

Claude Code (CLI) has been really bad with handling window resizing and layout changes. For long sessions, it will freeze while redrawing. On an M3 Max. With 64 GB or memory. Codex CLI does not have this problem, nor does OpenCode. Every time I split the pane, I'm dumbfounded by how bad this is.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#73

Post by atwrk »

TedDoesntTalk wrote: post #54
imtringued wrote: post #48

Earlier quoted context omitted.

This is correct. People seem to get the wrong idea about why agentic coding is even a thing in 2026. The naive AI techno optimist which has basically displaced the vast majority of opinions on HN, thinks that the models got "smarter" [0]. No, the training distribution shifted towards training on agentic sessions which made certain forms of agentic coding "in-distribution". We are still witnessing the same underlying…

You think the hugging face incident was a stunt? Can you explain?

OpenAI started fearmongering way back with GPT 2, arguing that model was too dangerous to release freely. That model was barely coherent enough for using it as a twitter bot. Anthropic just hopped onto that later. Conveniently, calling for regulation now would ease the competition from open Chinese models, opening the chance for both companies to eventually reach positive ROI, with consumers paying the price.

The burden of proof that this isn't just a publicity stunt again is squarely on them.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#74

Post by corv »

anon373839 wrote: post #59
corv wrote: post #11

My own findings are in line with this research: Having a coding harness is critical but the differences between them are overstated. Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes They’d all do the job - I just prefer to compartmentalize for access control. Keeping the harness’ surface area tiny had the added benefit of preser…

> Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes That's really interesting. I like Pydantic AI a lot and wondered why all of the harnesses seem to be written in Javascript instead of it. What do you use it for headless, though? I haven't tried Hermes or similar yet, so don't have a handle on what you do with them.

I initially went with OpenCode's v2 client-server rewrite until I noticed how that wasn't ready for prime time.

Turns out I didn't need a fancy TUI at all when desktop, web and mobile interfaces are already provided by or available for Hermes.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#75

Post by Otterly99 »

The most astonishing thing to me is that Pi harness is basically as efficient as the Codex/Claude.

I wonder if the same is true for the smaller models in the 9-32B range? I would expect that these models need more steering, but again I was not expecting this result either.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#76

Post by artdigital »

nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

Very curious as well to see comparisons between Claude Code, Codex, Pi, Grok, OpenCode, Crush, etc

Personally jumping around a lot to get a feeling for exactly that, and these days liking the Grok harness out of all of them the most

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#79

Post by kittikitti »

I find that in many cases, you ARE the harness. Case in point, Terence Tao utilized the simple chat interface to find the Jacobian Conjecture counterexample. This is arguable no harness at all. I also find that many people will disagree with me but most of the time, they just want a button to press that will solve the problem. If that's the work you'd like to accomplish, then the harness matters, but as the article alludes to, it doesn't need to be complex.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#80

Post by yearolinuxdsktp »

ygouzerh wrote: post #42
corv wrote: post #11

My own findings are in line with this research: Having a coding harness is critical but the differences between them are overstated. Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes They’d all do the job - I just prefer to compartmentalize for access control. Keeping the harness’ surface area tiny had the added benefit of preser…

One thing I am missing to be able to move out of Claude Code, is the auto mode (and the soft_deny and hard_deny settings that can be tuned), with it's classifier checking the output. It's the killer feature from me personally, often when wanting to troubleshoot for example things like Kubernetes workloads. LLMs are now really good at it, but we doesn't want them to like delete a pod. Other harnesses like Codex have o…

Codex has an “Approve for me” mode (“Guardian” internally) that does the same now.
Post reply on HN