Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

301–310 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#301

Earlier quoted context omitted.

> That’s not a threat. It’s free R&D. That's not a human. It's AI slop.

Yeah the article is full of it, especially the second half. I wonder if at any point we’ll be able to ban slop / low quality content from the internet, I don’t understand why this keeps getting upvoted.

It wouldn't even occur to me to submit ai slop to HN. Some people have no shame.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#302
post #204

Earlier quoted context omitted.

On macOS, Option+Shift+- and Option+- insert an em dash (—) and en dash (–), respectively. On Linux, you can hit the Compose Key and type --- (three hyphens) to get an em dash, or --. (hyphen hyphen period) for an en dash. Windows has some dumb incantation that you'll never remember.

For Windows it's just easier to make a custom keyboard layout and go to town with that: https://www.microsoft.com/en-us/download/details.aspx?id=102...

I prefer the linux compose-key style: https://github.com/samhocevar/wincompose

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#303
post #92

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

If I remember, both Claude Code and OpenAI Codex "harnesses" improved themselves now. OpenAI used early versions of GPT-5.3-Codex to: debug its own training process, manage its deployment and scaling and diagnose test results and evaluation data. Claude Code have shipped 22 PRs in a single day and 27 the day before, with 100% of the code in each PR generated entirely by Claude Code.

> with 100% of the code in each PR generated entirely by Claude Code.

You can tell...

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#304

Earlier quoted context omitted.

Okay, but why on earth should I as an OpenCode user accept that limitation when OpenAI explicitly supports 3rd party clients? That's how competition works in a healthy market. I certainly haven't built up enough brand loyalty to tolerate Anthropic's behavior as they tightened usage quotas on the Pro plan to the point of becoming unusable for actual development. (And sure, they probably don't care because they're losi…

> Anthropic's profit margin or lack thereof isn't my problem as a consumer when alternatives exist. For now. When a company is deliberately trying to be profitable and not subsidized by VC money; I'm more likely to buy their product. I have no desire to live further in a world run by monopolies.

But Anthropic very much is subsidized by VC money, just like OpenAI. They just raised another $30 billion this week. I'm not sure how Anthropic managed to position themselves as the "good guy" in many people's minds, but from my vantage they're much more similar to OpenAI than they are different.

So for the time being I'll stick with the option that isn't trying to profit from lock-in in the long term vs. competing on technical merits of their product. The great thing about basing a workflow on a tool like OpenCode is that if OpenAI enshittifies Codex, I don't have to worry about being trapped and can easily pivot to an open source model, or Anthropic via the API, etc depending on how the future turns out.

I've been paying for the ultra cheap Z.ai plan as my fallback for a while now anyway, or I can use my Github Copilot or Gemini AI Pro plan via OpenCode integrations (though the Gemini integration is probably the least stable of the four), so I certainly won't hesitate to drop OpenAI too if they give me sufficient cause.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#305

Earlier quoted context omitted.

Perhaps HN needs a guideline: "Suggesting that a comment was generated by an LLM without evidence adds little to a discussion and in fact deflects from the point being made. Please refrain from this."

Or a guideline that encourages users to downvote suggestions to police how other users think and communicate.

I love the irony of your post.

But I'm going to guess HN tried the no-rules approach and found issues with it. Whether I like them or not, there are rules and I often see others reminding us of them.

(Ha ha, and in point of fact, I have never read them except when one is trotted out. Nor have I ever pulled one on someone—I'm the type to ignore and move on.)

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#306
post #152

Earlier quoted context omitted.

tilth isn’t trying to replace grep for raw text search — for that, it wraps ripgrep internally so perf is comparable. It’s about reducing round-trips and giving the agent a verified edit workflow, not faster search. Instead of cat + grep + manual line counting, one tool call returns a structural outline of a large file, lets you drill into sections, and since this last update also returns hashline-anchored output tha…

well yah, that's what I mean how better is it versus cat + grep + manual line counting. Agents tend to perform worse with niche tools

It was really helpful to make and run a benchmark - it led to some important changes and improvements, so thanks again for your question kp!

The result is ~17% reduction in raw cost. If calculated per correct answer, its ~25% reduction per correct answer.

Just posted the update -> https://news.ycombinator.com/item?id=47016959

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#307

Earlier quoted context omitted.

Perhaps HN needs a guideline: "Suggesting that a comment was generated by an LLM without evidence adds little to a discussion and in fact deflects from the point being made. Please refrain from this."

Or a guideline that encourages users to downvote suggestions to police how other users think and communicate.

This is reminiscent of the early discussions around spam, before it was called that.

You might be surprised to learn the free speech side was crushed in that debate and it turned out content that costs zero to send really is worthless.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#308

When you're in the business of selling tokens - you look at technology that reduces that as a threat. If they were selling services that USE tokens, then reducing them would be welcome... so they'll likely steal this and incorporate it into their proprietary CLIs like claude code...

Huh? Anthropic doesn't sell Claude Code, they sell tokens. Why would they make Claude Code more token-efficient?

Anthropic sells api tokens - until they released claude code the only way to use claude for coding was via api tokens in something like cursor or cline.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#309
I made a benchmark on the top tier models as this is not the case in the article. Also I did several cases and the result is self speaking. Hashline is not an improvement that speaks for itself. It is the overhead of harness itself in tool descriptions and schemas. Here are my benchmark results and also my repo for a plugin to truely reduce token usage in top tier models: https://github.com/ASidorenkoCode/openslimedit

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#310

Earlier quoted context omitted.

Huh? Anthropic doesn't sell Claude Code, they sell tokens. Why would they make Claude Code more token-efficient?

Anthropic sells api tokens - until they released claude code the only way to use claude for coding was via api tokens in something like cursor or cline.

Exactly. They don't make money on Claude Code directly, so it's not in their interest to make it use fewer tokens (which are what they make their profit on).
Post reply on HN