Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

11–20 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#11

Post by theturtletalks »

woeirua wrote: post #6

The harness matters far more than most people think. This post about the CORE benchmark where Opus’ score almost doubled when they switched to Claude Code from their own harness. https://x.com/sayashk/status/1996334941832089732

Mario, the creator of Pi terminal agent, has this great blog post[0]. He talks about how TerminalBench's highest scores comes from using the Terminus 2 harness which uses tmux under the hood.

When I was reading the Opus 4.6 launch post, they mentioned the same thing and their TerminalBench score was based on using Terminus 2 and not CC.

0. https://mariozechner.at/posts/2025-11-30-pi-coding-agent/

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#12

Post by 1313ed01 »

rafaelmn wrote: post #5

I wonder if we'll get to "VI for LLMs" - if the model was trained on using that kind of text navigation and you show context around cursor when it navigates. Would also be worth having special tokens for this kind of navigation.

I always thought ed would be a perfect match. Line-based instead of having to manage cursor movements.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#14

Post by horsawlarway »

withinboredom wrote: post #7
woeirua wrote: post #6

The harness matters far more than most people think. This post about the CORE benchmark where Opus’ score almost doubled when they switched to Claude Code from their own harness. https://x.com/sayashk/status/1996334941832089732

Which, IMHO, should be why we should be able to change them freely or make our own. Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb.

Also another place where having it change out from underneath you can drastically alter the quality of your work in unexpected ways.

Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly.

Even if the "limits" on them stay generous, the product will start shifting to prioritize things the user doesn't want.

Tool recommendations are my immediate and near term fear - paid placement for dev tools both at the model level and the harness level seem inevitable.

---

The right route is open models and open harnesses, ideally on local hardware.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#15

Post by infecto »

animan wrote: post #13

What was the point of Claude code or Gemini banning the OP? Why would they care about how IDEs use the underlying API?

I assume he was using Gemini the same way as he was Claude when I make the following statement.

I don’t believe it’s exceptionally unique or new that companies will revoke access if you are using an unpublished API that the apps use. I don’t see anything wrong with it myself. If you want, pay for normal token use on the published APIs. There is no expectation that you can use APIs for an application, even if you are a paid user, that are not published explicitly for usage.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#16

Post by notsylver »

I feel like cursors solution is still the best answer. Let the model suggest edits in whatever format it prefers using as few "extra" tokens as possible and have a small model figure it out. I don't use cursor anymore but when I did it was impressive how consistently it worked, I think there was a single time it failed. 70b might be overkill though...

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#17

Post by deaux »

Great article, recommend reading all of it.

> Why bother, you ask? Opus may be a great model, but Claude Code to this day leaks raw JSONL from sub-agent outputs, wasting hundreds of thousands of tokens. I get to say, “fuck it, subagents output structured data now”.

This is why I find the banning of using Claude subscriptions in other harnesses is so heinous. Their harness that they're forcing onto everyone has tons of big issues including wasting massive numbers of tokens. Very much in line with intentionally refusing to adhere to standards in the most IE6 way possible.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#18

Post by kachapopopow »

My personal notes (not the author): have been way faster performance wise which is honestly the biggest improvement over correctless. I've posted https://github.com/can1357/oh-my-pi before, but didn't seem to gain traction. It's a great little agent.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#19

Post by eshaham78 »

horsawlarway wrote: post #14
withinboredom wrote: post #7

Earlier quoted context omitted.

Which, IMHO, should be why we should be able to change them freely or make our own. Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb.

Also another place where having it change out from underneath you can drastically alter the quality of your work in unexpected ways. Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly. Even if the "limits" on them stay generous, the product will start shifting to prioritize things the user doesn't want. Tool recommendations are my imm…

[dead]

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#20

Post by deaux »

horsawlarway wrote: post #14
withinboredom wrote: post #7

Earlier quoted context omitted.

Which, IMHO, should be why we should be able to change them freely or make our own. Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb.

Also another place where having it change out from underneath you can drastically alter the quality of your work in unexpected ways. Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly. Even if the "limits" on them stay generous, the product will start shifting to prioritize things the user doesn't want. Tool recommendations are my imm…

At this point subsidizing Chinese open-weights vendors by paying for them is just the right thing to do. Maybe they too might go closed-weights when they become SotA, but they're now pretty close and haven't done it.
Post reply on HN