Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

111–120 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#111

Post by benreesman »

The logical end state of this line of reasoning is a collective action problem that dooms the frontier lab establishment. You can't devote model capacity to having an attention transformer match nested delimiters or cope with bash and be maximally capable, you can't mix authentication, authorization, control plane, and data plane into an ill specified soup and be secure enough for any that isn't a pilot or toy ever.

If you run this out, you realize that the Worse is Better paradox has inverted, it's an arbitrage, and the race is on.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#112

Post by robotresearcher »

parhamn wrote: post #76

On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?

Humans have a demonstrated ability to program computers by flipping switches on the front panel.

Like a good programming language, a good harness offers a better affordance for getting stuff done.

Even if we put correctness aside, tooling that saves time and tokens is going to be very valuable.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#113

Post by ianbutler »

It’s funny to see where we are on model improvements.

Back when I was maintaining a coding harness around the time of Claude 3.5 we tried hash prefixes we tried line number prefixes we tried a lot of different approaches to making the model better at selecting edit blocks and ultimately at-least then fuzzy string matching won out.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#114

Post by senordevnyc »

robbomacrae wrote: post #63
tosh wrote: post #30

Shows how much room for improvement there is on the harness level. Agents waste a lot of tokens on editing, sandboxes, passing info back and forth from tool calls and subagents. Love the pragmatic mix of content based addressing + line numbers. Beautiful.

Indeed. The biggest waste might be the overuse of MCP for everything. Sure it makes the initial development easier but then for every connection you're using a hundred billion dollar parameter model to decide how to make the call when it's usually completely unnecessary and then prone to random errors. MCP is the hammer that can make literally everything look like a nail...

I see this ranting against MCP all the time, and I don't get it, maybe I'm missing something. I'm currently using an MCP in Cursor to give agents read-only access to my staging and prod databases, as well as BugSnag's MCP so it can look up errors that happen in those environments. It works great. What should I be using for this if not MCP?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#115

Post by senordevnyc »

Aurornis wrote: post #45
withinboredom wrote: post #7

Earlier quoted context omitted.

Which, IMHO, should be why we should be able to change them freely or make our own. Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb.

> Being locked into a specific harness because you pay 20 bucks per month vs. pay-per-use ... is kinda dumb. I’ll probably get downvoted for this, but am I the only one who thinks it’s kind of wild how much anger is generated by these companies offering discounted plans for use with their tools? At this point, there would be less anger and outrage on HN if they all just charged us the same high per-token rate and off…

No, you're not the only one. The outraged entitlement is pretty funny tbh. How dare they dictate that they'll only subsidize your usage if you use their software!!

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#117

Post by madeofpalk »

parhamn wrote: post #76

On first principles it would seem that the "harness" is a myth. Surely a model like Opus 4.6/Codex 5.3 which can reason about complex functions and data flows across many files would trip up over top level function signatures it needs to call? I see a lot of evidence to the contrary though. Anyone know what the underlying issue here is?

Isn't 'the harness' essentially just prompting?

It's completely understandable that prompting in better/more efficient means would produce different results.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#119

Post by visarga »

Yeah I invented a similar method for information extraction attribution around 2022, I would place custom markers in a document so the extraction model can reference them together with the answer and be unique on the document to be able to locate it.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#120

Post by ahofmann »

kachapopopow wrote: post #98
logicprog wrote: post #86

Earlier quoted context omitted.

Because I like them?

reminds me of that one guy complaining that everyone is calling them an AI when AI was trained on their grammar style.

This happened to the female speaker with her voice, which I find terrifying: https://www.youtube.com/watch?v=qO0WvudbO04
Post reply on HN