Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

261–270 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#261

Post by vohk »

theanonymousone wrote: post #252

Non-native speaker here. Can someone please be so nice to explain why do we use the word "Harness" here and not e.g. Orchestrate or Steer? It took me some time to realise what people mean by it, originally confusing it with harvest.

As already mentioned, this is the noun use but also different connotations.

To my thinking, to orchestrate or steer suggests a conductor or driver, an outside entity providing direction. A master agent creating and directing subagents could reasonably be called an orchestrator.

A harness is what the horse wears to pull a cart, or what connects a pilot to a parachute and provides the controls to tug on and steer. It might provide guidance or capability, but not active direction. It's also a fairly common use in hardware ( a wire harness) and software (a testing harness) already.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#264

Post by bradfa »

woah wrote: post #170

Seems like a very cool technique, but also very oversold. He's seeing a 5% improvement on a find and replace benchmark of his own devising and saying stuff like this in the blog post: > Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D. He makes it sounds like he got a 5-14% boos…

Sure but if we find another few “easy” 5% improvements in find/replace/edit (which is one of the most important actions for coding) then they really start to add up.

Most harnesses already have rather thorough solutions for this problem but new insights are still worth understanding.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#265

Post by subscribed »

fazgha wrote: post #71
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

[flagged]

AltGr+-

Why, can't make it work?

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#266

Post by int_19h »

JSR_FDED wrote: post #227
vardalab wrote: post #163

Earlier quoted context omitted.

Yeah, I agree with you. I'm so tired of people complaining about AI-generated text without focusing on the content. Just don't read it if you don't like it. It's another level of when people complain how a website is not readable for them or some CSS rendering is wrong or whatever. How does it add to the discussion?

The problem is that there’s infinite “content” out there. The amount of work the author puts in is correlated with the value of the piece (insight/novelty/etc). AI-written text is a signal that there’s less less effort and therefore less value there. It’s not a perfect correlation and there are lots of exceptions like foreign language speakers, but it is a signal.

Those who are convinced that every other poster is secretly AI can just not engage with those comments then.

As it is, it just adds noise. Much more so than AI-written comments themselves, at least here on HN.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#267

Post by athrowaway3z »

athrowaway3z wrote: post #183
woah wrote: post #170

Seems like a very cool technique, but also very oversold. He's seeing a 5% improvement on a find and replace benchmark of his own devising and saying stuff like this in the blog post: > Here is why that is backwards. I just showed that a different edit format improves their own models by 5 to 14 points while cutting output tokens by ~20%. That’s not a threat. It’s free R&D. He makes it sounds like he got a 5-14% boos…

> “replace line 2:f1, replace range 1:a3 through 3:0e, insert after 3:0e.” Not sure what they're calculating, but this seems to me like it could be many times more efficient than 20%.

So i just build this - with a few changes to the approach and usable as a simple pi-extention without having to use what-the-pi. It seems to work pretty well so far.

https://github.com/offline-ant/pi-hh-read

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#268

Post by benterix »

andai wrote: post #254
jstummbillig wrote: post #206

Earlier quoted context omitted.

The problem with these exercises is always: I have limited time and capacity to do things, and a fairly unlimited number of problems that I can think of to solve. Coding is not a problem I want to solve. Prompt engineering is not a problem I want to solve. If I do things for the love if it, the rules are different of course. But otherwise I will simply always accept that there are many things that improve around me,…

Well it's an amusing exercise I suppose, if you're into that sort of thing. I certainly enjoy it! My meaning, rather, is that there's people whose full time job is to build these things who seem to have forgotten what everyone in the field knew 3 years ago. More likely they think, ahh we don't need that now! These are all solved problems! In my experience, that's not really true. The stuff that worked 3 years ago sti…

> My meaning, rather, is that there's people whose full time job is to build these things who seem to have forgotten what everyone in the field knew 3 years ago.

Well, sometimes I wonder if this is actually true. I have an unprovable feeling that (1) some people do things that work better but they keep it to themselves, (2) some companies could do better wrt the optimization of the number of tokens getting it or out but they deliberately chose not to.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#269

Post by antihero »

kachapopopow wrote: post #18

My personal notes (not the author): have been way faster performance wise which is honestly the biggest improvement over correctless. I've posted https://github.com/can1357/oh-my-pi before, but didn't seem to gain traction. It's a great little agent.

But will you get banned for using it?
Post reply on HN