Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

blog.can.ac

Post reply on HN
Print view View original

251–260 of 318 posts

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#251

Post by lbreakjai »

georgemcbay wrote: post #237
spprashant wrote: post #233

It seems like agentic (or atleast AI-assisted coding) is the future. And we will be increasingly relying on these models to earn our livelihood. Is anyone else worried at how easily Anthropic/Google/OpenAI can basically cut you off if you do something they don't like?

> Is anyone else worried at how easily Anthropic/Google/OpenAI can basically cut you off if you do something they don't like? Yeah, had that thought here a few weeks ago on HN after reading about someone getting cut off from Claude: https://news.ycombinator.com/item?id=46723384#46728649 Though tbh I'm far more worried about the societal impacts of large scale job displacement across so many professional industries at…

I stopped worrying. Western societies have about 30 to 40% of the people doing knowledge work, which contributes to the economy that employs the other 60%.

If that 40% is automated away in one go, there's no economy as we know it anymore. Either it acts as a negative void coefficient and moderates it into something sustainable, or it blows up.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#252

Post by theanonymousone »

Non-native speaker here. Can someone please be so nice to explain why do we use the word "Harness" here and not e.g. Orchestrate or Steer?

It took me some time to realise what people mean by it, originally confusing it with harvest.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#253

Post by energy123 »

theanonymousone wrote: post #252

Non-native speaker here. Can someone please be so nice to explain why do we use the word "Harness" here and not e.g. Orchestrate or Steer? It took me some time to realise what people mean by it, originally confusing it with harvest.

Well, "Orchestrate" and "Steer" are verbs, while "Harness" is a noun. You need a noun here, not a verb, because the harness is not actively doing anything, it's just a set of constraints and a toolset.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#254

Post by andai »

jstummbillig wrote: post #206
andai wrote: post #178

Earlier quoted context omitted.

My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…

The problem with these exercises is always: I have limited time and capacity to do things, and a fairly unlimited number of problems that I can think of to solve. Coding is not a problem I want to solve. Prompt engineering is not a problem I want to solve. If I do things for the love if it, the rules are different of course. But otherwise I will simply always accept that there are many things that improve around me,…

Well it's an amusing exercise I suppose, if you're into that sort of thing. I certainly enjoy it!

My meaning, rather, is that there's people whose full time job is to build these things who seem to have forgotten what everyone in the field knew 3 years ago.

More likely they think, ahh we don't need that now! These are all solved problems! In my experience, that's not really true. The stuff that worked 3 years ago still works, and much of it works better.

Some of it doesn't work, for example, if the codebase is very large, but that's not difficult to account for. Poking around blindly, I say, should be the fallback in such cases, rather than the default in all of them!

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#255

Post by andai »

andai wrote: post #191
logicprog wrote: post #186

Earlier quoted context omitted.

> Well, surely none of this is relevant today? Well, it turns out all of it still is! e.g. small fix, the "grep def" (or your language's equivalent) can be trivially added as a startup hook to Claude Code, and suddenly it doesn't have to spend half your token budget poking around the codebase, because -- get this -- it can just see where everything is... (What a concept, right?) Hahaha yeah. This is very true. I find…

To clarify, I didn't mean we should be using ancient models in production, I meant in R&D. Anthropic says "do the simplest thing that works." If it works with the LLMs we had 3 years ago, doesn't that make it simpler? The newer LLMs mostly seem to work around the poor system design. (Like spawning 50 subagents on a grep-spree because you forgot to tell it where anything is...) But then you get poor design in prod!

As an addendum... The base/text models which have fallen out of style, are also extremely worth learning and working with. Davinci is still online, I believe, although it is deprecated.

Another lost skill! Learning how things were done before instruct tuning forces you to structure things in such a way so the model can't do it wrong. Half a page of well crafted examples can beat 3 pages of confusing rules!

(They're also magical and amazing at writing, although they produce bizarre and horrifying output sometimes.)

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#256

Post by viraptor »

energy123 wrote: post #253
theanonymousone wrote: post #252

Non-native speaker here. Can someone please be so nice to explain why do we use the word "Harness" here and not e.g. Orchestrate or Steer? It took me some time to realise what people mean by it, originally confusing it with harvest.

Well, "Orchestrate" and "Steer" are verbs, while "Harness" is a noun. You need a noun here, not a verb, because the harness is not actively doing anything, it's just a set of constraints and a toolset.

That doesn't really answer the questions, because there's orchestrator and steering.

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#258

Post by planckscnst »

There is so much work we can do with harnesses that can make the already existing models so much more capable. I definitely feel the author's frustration as I've also been working on some harness stuff. When Anthropic subscriptions got cut off from OpenCode and other third party tools, I was very disappointed because the model I do the most work in is Claude and I was specifically developing a change [1] in the hopes it would make Claude even better. After that, I started implementing the feature in Claude Code directly (using tweakcc) and after a day of working on that, they even block my tweaked Claude Code with the same message. It means I simply won't be able to use this idea with Claude at all

[1]: the README.md describes the Context Bonsai features in my fork here: https://github.com/Vibecodelicious/opencode

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#259

Post by mikodin »

theanonymousone wrote: post #252

Non-native speaker here. Can someone please be so nice to explain why do we use the word "Harness" here and not e.g. Orchestrate or Steer? It took me some time to realise what people mean by it, originally confusing it with harvest.

What always came to mind for me is an “engine wiring harness”. It’s responsible for getting power and data to all the right places without having to manually route cables around the engine / car.

If you google an image of it, maybe it’ll make sense

Re: Improving 15 LLMs at Coding in One Afternoon. Only the Harness Changed

#260

Post by grumbelbart »

andai wrote: post #178
logicprog wrote: post #65

I really enjoyed this article. I think the author is precisely right and I've been saying this for a long time. There's a ton of extremely interesting low hanging fruit that can vastly improve the effectiveness of even currently existing models hiding in how we design our agent harnesses; enough to — at least until we hit diminishing returns — make as much or more of a difference than training new models! I think one…

My Weird Hill is that we should be building things with GPT-4. I can say unironically that we haven't even tapped the full potential of GPT-4. The original one, from 2023. With no reasoning, no RL, no tool calling, no structured outputs, etc. (No MCP, ye gods!) Yes, it's possible to build coding agents with it! I say this because I did! Forcing yourself to make things work with older models forces you to keep things…

> My Weird Hill is that we should be building things with GPT-4.

Absolutely. I always advocate that our developers have to test on older / slower machines. That gives them direct (painful) feedback when things run slow. Optimizing whatever you build for an older "something" (LLM model, hardware) will make it excel on more modern somethings.

Post reply on HN