Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

211–220 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#211

Post by cyanydeez »

socketcluster wrote: post #208

Seeing all these 'coding' benchmarks reminds me that people still don't understand what coding means in practice. People still think one-phase puzzle-solving is coding. Real coding almost always has multiple phases which build on top of one another. There is an architectural component which is missed here - and the sheer number of phases/layers is actually where most of the complexity comes from.

Usually what I need a LLM to do is find me a elegant agorithm for a problem I've encountered where I know there's an elegant algorithm but I've got no idea what it's called or how to google search for it.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#212

Post by cyanydeez »

lillecarl wrote: post #110
TZubiri wrote: post #94

Earlier quoted context omitted.

But they rely on distilling the output of american leader models. Which will probably train against their own harness. Someone has to do the baseline training, development, and innovation. it can't be clones all the way down

Citation needed, SOTA labs surely has technical protection and legaleese against using them for training. It's been done in th past but what indicates this is still the case?

this didn't stop the millions of copyrighted works used to train the models.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#214

Post by cyanydeez »

HarHarVeryFunny wrote: post #166
babkayaga wrote: post #133

Still weird to me that most people are not just giving an LLM an access to an editor, forcing it to write shell scripts to edit files. Shrug.

That's not quite how it works, and anyways if the model can't generate an accurate find/replace string, why would you expect it to do any better generating accurate commands to drive your editor (assuming it knew how do do that in the first place) ?! The way edits happen is that the agent (local) first tells the model (typically remote) that it has an edit tool (e.g. taking parameters file name, find string and repla…

Seems like it's veering towards a per-model protocol similar to the expectation that these models will develop their own languages to speak among themselves as agents.

The trouble is though, because it's all indeterminant slop, every model will break in small ways that you're back to indeterminancy and building a harness ontop of the harness.

Still, , there's probably a way to get the local model and arbitrary remote model to agree on how to make a method call. But the only way that will be fruitful if you find a highly reproducible set of tuples within the model's shared space.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#215

Post by perrygeo »

Witness the giant leap forward in the capabilities of coding agents over the last year. There has been no such leap in LLM model performance. I think the causality is crystal clear. It's nothing about "AGI" and all about existing LLMs learning to use existing tools.

Even a sub-par LLM, put into a context where it has access to unix tools and network and files etc, is vastly more capable than the best LLM chatbot.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#216

Post by kristopolous »

I independently invented a very similar method and then abandoned it because it relies on abstraction.

Instead I now use Damerau-Levenshtein distance to assert the edits to be replaced and if the similarity is over some threshold the edit goes through

Really works well because it's explicit. Forcing the model to emit the source tokens to be replaced seems to improve things.

https://github.com/day50-dev/sidechat/blob/db9c8f9d834967442...

It will often chomp white space differently but the main problem is

1. Track alignment with the lines being tracks (hash fixes that)

2. Content alignment with the model not losing focus (hamming/levenshtein other similarity scores fixes that)

If we demand exact matches we're simply not going to get them.

(Combining both methods might be good, I hadn't thought of that)

Another crucial point: the error line "Content mismatch. Reread the file" is crucial. Errors should give descriptive remediate actions.

So even with crappy models it does this automatically and will tool loop accordingly.

Asking it to do smaller edits is no good. Many smaller models will go down to single line edits, looking around for blank lines and just inject garbage. So don't suggest it.

Larger models, which succeed in doing this, know to do that. Smaller models which don't, won't do it if you don't suggest it

Seriously this thing works with 4B models

I also combine it with a toolcall hack for models that don't support tool calling

https://github.com/day50-dev/sidechat/blob/db9c8f9d834967442...

It injects the tool description in the system prompt after probing the capabilities and then does a simple response router.

I haven't found a model within reason that this doesn't work with (I'm sure if you intentionally throw some fine tune botch up that's emitting garbage it'll break - that's not the claim)

YMMV, works for me™

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#217

Post by Aurornis »

TuxSH wrote: post #150
Aurornis wrote: post #44

Earlier quoted context omitted.

> Like most things - assume the "20/100/200" dollar deals that are great now are going to go down the enshitification route very rapidly. I don’t assume this at all. In fact, the opposite has been happening in my experience: I try multiple providers at the same time and the $20/month plans have only been getting better with the model improvements and changes. The current ChatGPT $20/month plan goes a very long way ev…

> The current ChatGPT $20/month plan goes a very long way It sure does and Codex is great, but do you think they'll maintain the current prices after/if it eventually dominates Claude Code in terms of marketshare and mindshare?

I think we'll always have multiple options providing similar levels of service, like we do with Uber and Lyft.

Unlike Uber and Lyft, the price of inference continues to go down as datacenter capacity comes online and compute hardware gets more powerful.

So I think we'll always have affordable LLM services.

I do think the obsession with prices of the entry-level plans is a little odd. $20/month is nothing relative to the salaries people using these tools receive. HN is full of warnings that prices are going to go up in the future, but what's that going to change for software developers? Okay, so my $20/month plan goes to $40/month? $60/month? That's still less than I pay for internet access at home.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#218

Post by kridsdale3 »

ex-aws-dude wrote: post #182
barrenko wrote: post #93

Earlier quoted context omitted.

2026 is the year of the harness.

As a VC in 2026 I'm going to be asking every company "but what's your harness strategy?"

Given that you're likely in San Francisco, make sure you say "AI Harness".

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#219

Post by senordevnyc »

canadiantim wrote: post #155
senordevnyc wrote: post #114

Earlier quoted context omitted.

I see this ranting against MCP all the time, and I don't get it, maybe I'm missing something. I'm currently using an MCP in Cursor to give agents read-only access to my staging and prod databases, as well as BugSnag's MCP so it can look up errors that happen in those environments. It works great . What should I be using for this if not MCP?

agent skills, or use claude code to iteratively condense an MCP you want to use into only its most essential tools for your workflow

Agent skills are just a markdown file, what’s in that markdown file in your scenario?

And the MCP already only has the most essential tools for my workflow: the ability to run queries against a few databases.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#220

Post by senordevnyc »

visarga wrote: post #144
senordevnyc wrote: post #114

Earlier quoted context omitted.

I see this ranting against MCP all the time, and I don't get it, maybe I'm missing something. I'm currently using an MCP in Cursor to give agents read-only access to my staging and prod databases, as well as BugSnag's MCP so it can look up errors that happen in those environments. It works great . What should I be using for this if not MCP?

Make a CLI tool for it, of course

What? Why? What advantage does that have over just using an MCP server that exposes tools to run queries?
Post reply on HN