Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

231–240 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#231

Post by theahura »

woah wrote: post #170

Seems like a very cool technique, but also very oversold. He's seeing a 5% improvement on a find and replace benchmark of his own devising and saying stuff like this in the blog post: > Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D. He makes it sounds like he got a 5-14% boos…

> That’s not a threat. It’s free R&D.

That's not a human. It's AI slop.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#232

Post by socketcluster »

cyanydeez wrote: post #211
socketcluster wrote: post #208

Seeing all these 'coding' benchmarks reminds me that people still don't understand what coding means in practice. People still think one-phase puzzle-solving is coding. Real coding almost always has multiple phases which build on top of one another. There is an architectural component which is missed here - and the sheer number of phases/layers is actually where most of the complexity comes from.

Usually what I need a LLM to do is find me a elegant agorithm for a problem I've encountered where I know there's an elegant algorithm but I've got no idea what it's called or how to google search for it.

It makes sense but if the goal is to replace software engineers as claimed, then these benchmarks aren't going to achieve that.

Companies are still stuck in this mindset conflating software engineering with puzzle-solving. This is evident from their job interviews and also these LLM benchmarks.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#233

Post by spprashant »

It seems like agentic (or atleast AI-assisted coding) is the future. And we will be increasingly relying on these models to earn our livelihood.

Is anyone else worried at how easily Anthropic/Google/OpenAI can basically cut you off if you do something they don't like?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#234

Post by senordevnyc »

nojito wrote: post #223
senordevnyc wrote: post #220

Earlier quoted context omitted.

What? Why? What advantage does that have over just using an MCP server that exposes tools to run queries?

Context. Why would I use an MCP when I can use a cli tool that the model likely trained on how to use?

Can you be more specific about “context”?

And not everything has a CLI, but in any case, the comment I was replying to was suggesting building my own CLI, which presumably the LLM wasn’t trained on.

Maybe my understanding of MCP is wrong, my assumption is that it’s a combination of a set of documented tools that the LLM can call (which return structured output), and a server that actually receives and processes those tool calls. Is that not right? What’s the downside?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#235

Post by TZubiri »

lillecarl wrote: post #110
TZubiri wrote: post #94

Earlier quoted context omitted.

But they rely on distilling the output of american leader models. Which will probably train against their own harness. Someone has to do the baseline training, development, and innovation. it can't be clones all the way down

Citation needed, SOTA labs surely has technical protection and legaleese against using them for training. It's been done in th past but what indicates this is still the case?

>Citation needed, SOTA labs surely has technical protection

They have unlimited APIs, as long as you pay, how would they control how you use them?

> and legaleese against using them for training.

It's a whole different jurisdiction, and in general chinese companies care way less about copyright infringement

https://en.wikipedia.org/wiki/Counterfeit_consumer_good https://en.wikipedia.org/wiki/Allegations_of_intellectual_pr... https://en.wikipedia.org/wiki/China%E2%80%93United_States_tr...

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#236

Post by gsb »

logicprog wrote: post #186
andai wrote: post #178

Earlier quoted context omitted.

My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…

> Well, surely none of this is relevant today? Well, it turns out all of it still is! e.g. small fix, the "grep def" (or your language's equivalent) can be trivially added as a startup hook to Claude Code, and suddenly it doesn't have to spend half your token budget poking around the codebase, because -- get this -- it can just see where everything is... (What a concept, right?) Hahaha yeah. This is very true. I find…

> A version of this that uses tree sitter grammars to map a codebase, and does it on every startup of an agent, would be awesome.

This was a key feature of aider and if you're not inclined to use aider (or the forked version cecli) I think a standalone implementation exist at https://github.com/pdavis68/RepoMapper

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#237

Post by georgemcbay »

spprashant wrote: post #233

It seems like agentic (or atleast AI-assisted coding) is the future. And we will be increasingly relying on these models to earn our livelihood. Is anyone else worried at how easily Anthropic/Google/OpenAI can basically cut you off if you do something they don't like?

> Is anyone else worried at how easily Anthropic/Google/OpenAI can basically cut you off if you do something they don't like?

Yeah, had that thought here a few weeks ago on HN after reading about someone getting cut off from Claude:

https://news.ycombinator.com/item?id=46723384#46728649

Though tbh I'm far more worried about the societal impacts of large scale job displacement across so many professional industries at the same time.

I think it is likely to be very, very ugly for society in the near term. Not because the problems are unsolvable, but because everyone is choosing to ignore the threat of them.

And I realize a lot of people will handwave my concerns away with stories of Luddites and Jevon's paradox, but we've never had a tidal wave this big hit all at once and I think the scale (combined with speed of change) fundamentally changes things this time.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#238

Post by storus »

codazoda wrote: post #222
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

Ive been working on Peen, a CLI that lets local Ollama models call tools effectively. It’s quite amateur, but I’ve been surprised how spending a few hours on prompting, and code to handle responses, can improve the outputs of small local models. https://github.com/codazoda/peen

Current LLMs use special tokens for tool calls and are thoroughly trained for that, nearing almost 100% correctness these days, allowing multiple tool calls per single LLM response. That's hard to beat with custom tool calls. Even older 80B models struggle with custom tools.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#239

Post by pmarreck »

andai wrote: post #178
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…

> semantic

> grep def

Once you get to a codebase beyond a certain size, that no longer works.

I've for one found Serena https://github.com/oraios/serena , which you can install from right within Claude, to be a fairly fantastic code-interaction tool for LLM's. Both semantic search as well as editing. And with way less token churn.

Post reply on HN