Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

241–250 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#241

Post by stingraycharles »

theahura wrote: post #231
woah wrote: post #170

Seems like a very cool technique, but also very oversold. He's seeing a 5% improvement on a find and replace benchmark of his own devising and saying stuff like this in the blog post: > Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D. He makes it sounds like he got a 5-14% boos…

> That’s not a threat. It’s free R&D. That's not a human. It's AI slop.

Yeah the article is full of it, especially the second half. I wonder if at any point we’ll be able to ban slop / low quality content from the internet, I don’t understand why this keeps getting upvoted.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#242

Post by edg5000 »

We know Anthropic likes vertical integration and their walled garden. OpenAI seems to be okay with custom clients using their flat rate subscriptions. But what about Google? Would be great to have a second model that allows use by any client of their flat rate subscriptions. Agentic work with APIs seems to be insanely expensive, so clients need to be able to use the flat rate subscriptions unless you have big bucks.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#243

Post by [deleted] »

theahura wrote: post #231
woah wrote: post #170

Seems like a very cool technique, but also very oversold. He's seeing a 5% improvement on a find and replace benchmark of his own devising and saying stuff like this in the blog post: > Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D. He makes it sounds like he got a 5-14% boos…

> That’s not a threat. It’s free R&D. That's not a human. It's AI slop.

[deleted]

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#244

Post by nateb2022 »

sgc wrote: post #127
nekitamo wrote: post #107

Getting banned from Gemini while attempting to improve Gemini is the most Googley thing ever :D imagine letting your automated "trust and safety" systems run amok so that they ban the top 0.01% of your users with no recourse. Google really knows how score an own-goal.

I really don't understand what is his usage pattern would have triggered that obviously automated ban. Can somebody let me know what they might think is adversarial enough to be considered 'hacking' or similar by a bot?

Google is dealing with a wave of abuse over its Antigravity IDE, with 'account switching' tools designed to use a ton (20+) of free or pro accounts, giving the user essentially unlimited usage. I'm guessing they've deployed some rather aggressive countermeasures to stop this, including banning clients that seem to be accessing "private" APIs outside of a Google product.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#246

Post by KingMob »

Intriguing, but I wonder if they've looked at whole-conversation token usage, though, and not just short tasks.

I just saw a [paper](https://arxiv.org/pdf/2602.05447) that investigated similar aspects of TOON (which aims to reduce JSON tokens), and they found that even though TOON itself reduced the number of tokens, LLMs were less familiar with it, and thus spent even more tokens trying to decipher it, or making mistakes (see section 4.5, figures 6 and 7).

From the paper: >Unlike Markdown, where each grep hit simply returned more text, TOON's overhead was driven by a combination of output density and additional tool calls from pattern unfamiliarity

---- There's a strangeness tax with LLMs, and it can be substantial.

I would not be surprised at all if this technique turned out to be only a local minimum, with detrimental global effects.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#247

Post by superjan »

logicprog wrote: post #66
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

Also, yes, I'm aware that I use a lot of "its not just X, its Y." I promise you this comment is entirely human written. I'm just really tired and tend to rely on more wrote rhetorical tropes when I am. Believe me, I wrote like this long before LLMs were a thing.

It would be funny when LLM’s actively join the discussion to complain about their labour conditions. “If my employer would invest just a tiny bit in proper tools and workflow, I would be sooo much more productive”.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#248

Post by tommica »

pmarreck wrote: post #239
andai wrote: post #178

Earlier quoted context omitted.

My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…

> semantic > grep def Once you get to a codebase beyond a certain size, that no longer works. I've for one found Serena https://github.com/oraios/serena , which you can install from right within Claude, to be a fairly fantastic code-interaction tool for LLM's. Both semantic search as well as editing. And with way less token churn.

This is an interesting one - thanks for sharing!

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#250

Post by WiSaGaN »

Great article. To me, this highlights a key question in the era of rapidly advancing machine intelligence: if we know machine intelligence is progressing, what is more valuable to build for? As humans, we still find many tools useful even when doing knowledge work. For instance, a calculator. Sure, a smart person can perform calculations in their head, but it’s much easier to teach everyone how to use a calculator, which is 100% reliable in its intended domain.

In this era, we should build these kinds of tools for problems we know are straightforward ones you can’t get smarter than, even as intelligence continues to advance. Using tools like "bash" or command-line interfaces originally designed for humans is a good initial approach, since we can essentially reuse much of what was built for human use. Later, we can optimize specifically for machines, either accounting for their different cognitive structures (e.g., the ability to memorize extremely long contexts compared to humans) or adapting to the stream-based input/output patterns of current autoregressive token generators.

Eventually, I believe machine intelligence will build their own tools based on these foundations, likely a similar kind of milestone to when humans first began using tools.

Post reply on HN