Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

121–130 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#121

Post by skydhash »

cruffle_duffle wrote: post #64
znnajdla wrote: post #40

My experience as well. People worry our profession is being reduced to "prompt engineer", but actually I get the feeling that programming will soon be mainly about designing and building harnesses for specific tasks.

The more I dive into this space the more I think that developers will still be in heavy demand—just operating in a different level of abstraction most of the time. We will need to know our CS fundamentals, experience will still matter, juniors will still be needed. It’s just that a lot of time time the actual code being generated will come from our little helper buddies. But those things still need a human in the sea…

> LLMs are just another way to talk to a machine. They aren’t magic.

I will still opt for a scriptable shell. A few scripts, and I have a custom interface that can be easily composed. And could be run on a $100 used laptop from ebay.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#122

Post by techpression »

deaux wrote: post #38
techpression wrote: post #28

Earlier quoted context omitted.

I mean they want to make money right? CC is a cool tool, but obviously they want you to use the api eventually if you’re even remotely a power user, 200/month for all you can eat tokens (well, until some arbitrary limit of the day kicks in) just doesn’t make sense when compared to api prices. In other words, CC should be seen as a software subscription.

The token limit is the same whether used in CC or in other harnesses.

Sure, but then Anthropic loses the possibility to upsell, show ads, telemetry, brag about number of users and how long they use it etc etc. Not necessarily what’s in there today, but what can be in there tomorrow. They also get the ability to much better fine tune backoffs etc from a purely technical side of things.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#123

Post by miohtama »

barrenko wrote: post #93
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

2026 is the year of the harness.

But will harness build desktop Linux for us?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#124

Post by azinman2 »

renewiltord wrote: post #109
azinman2 wrote: post #99

Why not just use line numbers?

Forces you to read after every write. E.g. you edit line 15 to be two lines. Then now you need arithmetic for later vs earlier lines or you need to read full file to reindex by line number.

Good point!

I just wonder how unique these hashes will be if only 2 characters. It seems like the collision rate would be really high.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#125

Post by visarga »

barrenko wrote: post #93
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

2026 is the year of the harness.

Already made a harness for Claude to make R/W plans, not write once like they are usually implemented. They can modify themselves as they work through the task at hand. Also relying on a collection of patterns for writing coding task plans which evolves by reflection. Everything is designed so I could run Claude in yolo-mode in a sandbox for long stretches of time.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#126

Post by dimgl »

chrisweekly wrote: post #36

Great post. A few choice quotes: > Often the model isn’t flaky at understanding the task. It’s flaky at expressing itself. You’re blaming the pilot for the landing gear. > The model is the moat. The harness is the bridge. Burning bridges just means fewer people bother to cross. Treating harnesses as solved, or even inconsequential, is very short-sighted. > The gap between “cool demo” and “reliable tool” isn’t model m…

My personal favorite: That’s not a threat. It’s free R&D.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#127

Post by sgc »

nekitamo wrote: post #107

Getting banned from Gemini while attempting to improve Gemini is the most Googley thing ever :D imagine letting your automated "trust and safety" systems run amok so that they ban the top 0.01% of your users with no recourse. Google really knows how score an own-goal.

I really don't understand what is his usage pattern would have triggered that obviously automated ban. Can somebody let me know what they might think is adversarial enough to be considered 'hacking' or similar by a bot?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#129

Post by jbellis »

ianbutler wrote: post #113

It’s funny to see where we are on model improvements. Back when I was maintaining a coding harness around the time of Claude 3.5 we tried hash prefixes we tried line number prefixes we tried a lot of different approaches to making the model better at selecting edit blocks and ultimately at-least then fuzzy string matching won out.

Yes, very similar results here (http://brokk.ai)

We got lines-with-anchors working fine as a replacement strategy, the problem was that when you don't make the model echo what it's replacing, it's literally dumber at writing the replacement; we lost more in test failures + retries than we gained in faster outputs.

Makes sense when you think about how powerful the "think before answering" principle is for LLMs, but it's still frustrating

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#130

Post by furyofantares »

madeofpalk wrote: post #117
parhamn wrote: post #76

On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?

Isn't 'the harness' essentially just prompting? It's completely understandable that prompting in better/more efficient means would produce different results.

No, it's also a suite of tools beyond what's available in bash, tailored to context management.
Post reply on HN