Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

181–190 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#181
post #71

Earlier quoted context omitted.

[flagged]

Does your friend have an iPhone? The default iOS keyboard has automatically converted double dashes into an emdash for at least seven years now.

I think Google docs does this too, which drives me up the wall when I'm trying to write `command --foo=bar` and it turns it into an M-dash which obviously doesn't work.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#182

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

2026 is the year of the harness.

As a VC in 2026 I'm going to be asking every company "but what's your harness strategy?"

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#183
post #170

Seems like a very cool technique, but also very oversold. He's seeing a 5% improvement on a find and replace benchmark of his own devising and saying stuff like this in the blog post: > Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D. He makes it sounds like he got a 5-14% boos…

> “replace line 2:f1, replace range 1:a3 through 3:0e, insert after 3:0e.”

Not sure what they're calculating, but this seems to me like it could be many times more efficient than 20%.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#185
post #69

Earlier quoted context omitted.

Once you begin to see the “model” as only part of the stack, you begin to realize that you can draw the line of the system to include the user as well. That’s when the future really starts hitting you.

[flagged]

> the user inclusion part is real too. the best results i get aren't from fully autonomous agents, they're from tight human-in-the-loop cycles where i'm steering in real time. the model does the heavy lifting, i do the architectural decisions and error correction. feels more like pair programming than automation.

Precisely. This is why I use Zed and the Zed Agent. It's near-unparalleled for live, mind-meld pair programming with an agent, thanks to CRDTs, DeltaDB, etc. I can elaborate if anyone is interested.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#186
post #178

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…

> Well, surely none of this is relevant today? Well, it turns out all of it still is! e.g. small fix, the "grep def" (or your language's equivalent) can be trivially added as a startup hook to Claude Code, and suddenly it doesn't have to spend half your token budget poking around the codebase, because -- get this -- it can just see where everything is... (What a concept, right?)

Hahaha yeah. This is very true. I find myself making ad hoc versions of this in static markdown files to get around it. Just another example of the kind of low hanging fruit harnesses are leaving on the table. A version of this that uses tree sitter grammars to map a codebase, and does it on every startup of an agent, would be awesome.

> My Weird Hill is that we should be building things with GPT-4.

I disagree, IMO using the best models we have is a good way to avoid wasting time, but that doesn't mean we shouldn't also be frugal and clever with our harnesses!

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#187
post #71

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

[flagged]

I just type -- and jira fixes it.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#188
When you're in the business of selling tokens - you look at technology that reduces that as a threat. If they were selling services that USE tokens, then reducing them would be welcome... so they'll likely steal this and incorporate it into their proprietary CLIs like claude code...

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#189
post #71

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

[flagged]

On Windows it is Alt+0151. Harder to use than on Mac but definitely possible, I frequently use it.

On recent versions Shift+Win+- also work, and Win+- produces en dash.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#190
post #152

Earlier quoted context omitted.

tilth isn’t trying to replace grep for raw text search — for that, it wraps ripgrep internally so perf is comparable. It’s about reducing round-trips and giving the agent a verified edit workflow, not faster search. Instead of cat + grep + manual line counting, one tool call returns a structural outline of a large file, lets you drill into sections, and since this last update also returns hashline-anchored output tha…

well yah, that's what I mean how better is it versus cat + grep + manual line counting. Agents tend to perform worse with niche tools

Thank you for this question - I'm building out a benchmark now. Initial results are very promising, will update you once it's done!
Post reply on HN