Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

291–300 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#291

Post by JKCalhoun »

logicprog wrote: post #66
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

Also, yes, I'm aware that I use a lot of "its not just X, its Y." I promise you this comment is entirely human written. I'm just really tired and tend to rely on more wrote rhetorical tropes when I am. Believe me, I wrote like this long before LLMs were a thing.

Perhaps HN needs a guideline:

"Suggesting that a comment was generated by an LLM without evidence adds little to a discussion and in fact deflects from the point being made. Please refrain from this."

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#292

Post by spyder »

Yea, LLMs have prompt-, harness- and even random seed variability, and it leaves you wonder maybe with a better prompt or system instruction a model could perform better. Too bad most benchmarks don't report that variability, because it could reveal that the model may only perform well if it's prompted in the style of their training data and not generalize well to unseen prompt styles. Also it could explain some of the benchmark vs real world usage gaps.

I remember some papers about earlier models having around 15% prompt variability, and with different tool use sometimes there are even more significant jumps. And if I remember correctly the reasoning models improve some of these because lot of the early prompting tricks is included in them like "thinking step-by-step", "think carefully" and some other "magic" methods. Also another trick is to ask the models to rephrase the prompt with their own words because that may produce prompt that better align with their training prompts. For sure the big model developers are aware of these and constantly improving it, I just don't see too much discussion or numbers about it.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#294

Post by toraway »

Aurornis wrote: post #45
withinboredom wrote: post #7

Earlier quoted context omitted.

Which, IMHO, should be why we should be able to change them freely or make our own. Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb.

> Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb. I’ll probably get downvoted for this, but am I the only one who thinks it’s kind of wild how much anger is generated by these companies offering discounted plans for use with their tools? At this point, there would be less anger and outrage on HN if they all just charged us the same high per-token rate and off…

Okay, but why on earth should I as an OpenCode user accept that limitation when OpenAI explicitly supports 3rd party clients? That's how competition works in a healthy market.

I certainly haven't built up enough brand loyalty to tolerate Anthropic's behavior as they tightened usage quotas on the Pro plan to the point of becoming unusable for actual development.

(And sure, they probably don't care because they're losing money on that plan but again, OpenAI offers a plan at the same price point with vastly superior usage limits so I just canceled Claude, subscribed to Codex and moved on with my life. Anthropic's profit margin or lack thereof isn't my problem as a consumer when alternatives exist.)

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#295

Post by toraway »

senordevnyc wrote: post #115
Aurornis wrote: post #45

Earlier quoted context omitted.

> Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb. I’ll probably get downvoted for this, but am I the only one who thinks it’s kind of wild how much anger is generated by these companies offering discounted plans for use with their tools? At this point, there would be less anger and outrage on HN if they all just charged us the same high per-token rate and off…

No, you're not the only one. The outraged entitlement is pretty funny tbh. How dare they dictate that they'll only subsidize your usage if you use their software!!

Yes, I'm entitled because I didn't stick around paying for a subpar plan compared to their direct competitor OpenAI who supports my use case at the same price point.

Any reasonable person should be thanking Dario for lock-in that protects us from nefarious alternative clients and pledging to pay even more for the privilege of lower usage limits!

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#296

Post by withinboredom »

toraway wrote: post #294
Aurornis wrote: post #45

Earlier quoted context omitted.

> Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb. I’ll probably get downvoted for this, but am I the only one who thinks it’s kind of wild how much anger is generated by these companies offering discounted plans for use with their tools? At this point, there would be less anger and outrage on HN if they all just charged us the same high per-token rate and off…

Okay, but why on earth should I as an OpenCode user accept that limitation when OpenAI explicitly supports 3rd party clients? That's how competition works in a healthy market. I certainly haven't built up enough brand loyalty to tolerate Anthropic's behavior as they tightened usage quotas on the Pro plan to the point of becoming unusable for actual development. (And sure, they probably don't care because they're losi…

> Anthropic's profit margin or lack thereof isn't my problem as a consumer when alternatives exist.

For now. When a company is deliberately trying to be profitable and not subsidized by VC money; I'm more likely to buy their product. I have no desire to live further in a world run by monopolies.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#297

Post by andai »

spyder wrote: post #292

Yea, LLMs have prompt-, harness- and even random seed variability, and it leaves you wonder maybe with a better prompt or system instruction a model could perform better. Too bad most benchmarks don't report that variability, because it could reveal that the model may only perform well if it's prompted in the style of their training data and not generalize well to unseen prompt styles. Also it could explain some of t…

I haven't been able to find it again, but a few years ago I read a paper that found that certain prompts massively improved the performance of some LLMs on benchmarks. But the same prompt massively reduced the performance of some other LLMs. I assume this is still true, though perhaps not as dramatically as before.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#298

Post by waffletower »

JKCalhoun wrote: post #291
logicprog wrote: post #66

Earlier quoted context omitted.

Also, yes, I'm aware that I use a lot of "its not just X, its Y." I promise you this comment is entirely human written. I'm just really tired and tend to rely on more wrote rhetorical tropes when I am. Believe me, I wrote like this long before LLMs were a thing.

Perhaps HN needs a guideline: "Suggesting that a comment was generated by an LLM without evidence adds little to a discussion and in fact deflects from the point being made. Please refrain from this."

Or a guideline that encourages users to downvote suggestions to police how other users think and communicate.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#299

Post by antonok »

rafaelmn wrote: post #5

I wonder if we'll get to "VI for LLMs" - if the model was trained on using that kind of text navigation and you show context around cursor when it navigates. Would also be worth having special tokens for this kind of navigation.

I had the same thought too. It's probably not too difficult to fine-tune a small model for it using the "introduce a random mutation and describe the issue" workflow from TFA

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#300

Post by antonok »

christophilus wrote: post #171

Has any harness matched the effectiveness of Claude Code yet? I haven't experimented much recently, but every time I have in the past, I wasn't able to get any other tool to approach how effective I am in CC. I'd love to use a different harness-- ideally an OSS one-- and hook it up to whichever LLM provides the best bang for the buck rather than being tied to Claude.

OpenCode has been great in my experience. I still get the best results using it with Anthropic's models, but some of the open weights ones are catching up (GLM 5 works reasonably well for me).
Post reply on HN