Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HarnessTax: How Much Does the Harness Matter for Coding Agents?

harnesstax.github.io

Post reply on HN
Print view View original

81–90 of 97 posts

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#81

Post by verdverm »

nojs wrote: post #23

We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models. I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count? My experience is that the harness is mainly polish preventing faile…

Coming Soon: https://artificialanalysis.ai/agents/coding-agents?coding-ag...

They used to have one chart with {model X harness} for a subset of combos, looks like that is getting an upgrade

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#82

Post by verdverm »

mobelkh wrote: post #57
calgoo wrote: post #39

Earlier quoted context omitted.

I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed. The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness. The big…

what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?

look into the prompts, there are some choices in there

disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi

pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#83

Post by esperent »

verdverm wrote: post #82
mobelkh wrote: post #57

Earlier quoted context omitted.

what does it do too much out if the box? afaik it only has the basic tools and two agents profiles, maybe lsp support is extra?

look into the prompts, there are some choices in there disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level

> look into the prompts, there are some choices in there

Such as?

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#84

Post by verdverm »

esperent wrote: post #83
verdverm wrote: post #82

Earlier quoted context omitted.

look into the prompts, there are some choices in there disclaimer, I use opencode and have customized parts of it, and will do more, but it is a solid foundation and comes with more out of the box than pi pi is too minimal for me, I'd go back to my custom built harness if I wanted to be back at that level

> look into the prompts, there are some choices in there Such as?

Instructing to use the `gh` CLI, not ideal for those of use using BBC at work

They also have instructions about how to format certain output, which conflicts with the instructions we have in repo. I only discovered yesterday because we were wondering why the agent kept picking certain tools.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#85

Post by nottorp »

atwrk wrote: post #73
TedDoesntTalk wrote: post #54

Earlier quoted context omitted.

You think the hugging face incident was a stunt? Can you explain?

OpenAI started fearmongering way back with GPT 2, arguing that model was too dangerous to release freely. That model was barely coherent enough for using it as a twitter bot. Anthropic just hopped onto that later. Conveniently, calling for regulation now would ease the competition from open Chinese models, opening the chance for both companies to eventually reach positive ROI, with consumers paying the price. The bur…

[dead]

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#87

Post by jedbrooke »

kittikitti wrote: post #79

I find that in many cases, you ARE the harness. Case in point, Terence Tao utilized the simple chat interface to find the Jacobian Conjecture counterexample. This is arguable no harness at all. I also find that many people will disagree with me but most of the time, they just want a button to press that will solve the problem. If that's the work you'd like to accomplish, then the harness matters, but as the article a…

I like to say with LLMs the real intelligence was inside us all along.

That said, system prompt (which to your point also comes from the human) matters a lot too. For example a coding harness that doesn’t a have a plan mode will keep trying to write underbaked ideas to code instead of working with the user on the plan. That said, a “system prompt” can also be the first prompt you send to the agent in a vanilla chat window, so I do like your point about the user being the real harness

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#88

Post by mikeocool »

vb-8448 wrote: post #86

The problem is that Pi, on any third party harness, cannot really compete with codex or cc due to subscriptions.

I believe you can use your codex subscription with Pi, and it’s officially endorsed by OpenAI and uses your subscription usage.

Though for cc you are correct, using it with your sub draws from extra usage.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#89

Post by logicchains »

vb-8448 wrote: post #86

The problem is that Pi, on any third party harness, cannot really compete with codex or cc due to subscriptions.

Deepseek harness can if using Deepseek V4.1 Flash from the official API, it's cheap enough to be competitive even without a subscription.

Re: HarnessTax: How Much Does the Harness Matter for Coding Agents?

#90

Post by wyre »

vb-8448 wrote: post #86

The problem is that Pi, on any third party harness, cannot really compete with codex or cc due to subscriptions.

Pi supports OpenAI login for ChatGPT subscriptions.

Every 3rd party open-source harness I know of supports ChatGPT subscriptions.

Post reply on HN