Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harness engineering for self-improvement

lilianweng.github.io

Post reply on HN
Print view View original

81–90 of 96 posts

Re: Harness engineering for self-improvement

#81

Post by jagenabler2 »

AlexErrant wrote: post #56
bisonbear wrote: post #41

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…

I literally hold a "retro" with my agent after a session. > Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in. It…

I’m just holding out for when this stuff becomes mainstream and automatically built into tooling. I don’t want to hold a retro with my agent. I actually want to spend as little time as possible talking with it.

Re: Harness engineering for self-improvement

#82

Post by megadragon9 »

I did something similar. It started off as a "self-improving agent" project, inspired by autoresearch, then later on I reframed it as "harness training" (discrete program search) borrowing the mental model from ML training.

I "trained" the harness on a subset of Terminal-Bench 2.0 tasks while keeping the LLM (local Qwen3.6-35B A3B) frozen. Making LLM inference and the task environments fully deterministic was necessary for clean credit assignment. I learned this the hard way after spending the initial 1 month on experiment noise.

My final results showed that on the full 89-task Terminal-Bench 2.0 suite, the trained harness matched or beat the official Terminus 2 harness for four LLMs that it never collaborated with during training (e.g. GPT-OSS-120B score increased from 18.7% to 36%, while using 55% fewer input tokens per solve). A harness trained only on SWE-bench improved Terminal-Bench scores too. Here's the write-up: https://www.henrypan.com/blog/2026-07-18-harness-training/

I packaged the training loop as a PyTorch-style framework. https://github.com/workofart/harness-training

Re: Harness engineering for self-improvement

#83

Post by AlexErrant »

jagenabler2 wrote: post #81
AlexErrant wrote: post #56

Earlier quoted context omitted.

I literally hold a "retro" with my agent after a session. > Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in. It…

I’m just holding out for when this stuff becomes mainstream and automatically built into tooling. I don’t want to hold a retro with my agent. I actually want to spend as little time as possible talking with it.

Sure, you can "vibe/auto-retro". But in my experience the retro often results in "higher-order" structural changes.

In starker terms: if you're ok with "vibe principles", "vibe invariants", "vibe lint-rules", "vibe skill updates", and "vibe agent-tooling"... well. That's your call.

Thankfully after a dozen or so retros the low hanging fruit starts to disappear. But until then, you've no idea what's there.

Re: Harness engineering for self-improvement

#84

Post by radlad »

GrinningFool wrote: post #76
radlad wrote: post #59

Earlier quoted context omitted.

Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable? As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible. My CLAUDE.md, memories, and skills are all about either (a) adherence to project standa…

You don't have to remove them - you can move things that are not important to every prompt to aseparate small docs that are referenced in agents.md with ,"when needed, reference these files:" and list them in form * relpath - content summary" or similar. . When you want those instructions to be followed, mentiont them in your prompt. "Test this following procedures on docs/test.md". (I've found I don't have to do tha…

That's what MEMORY.md and subdirectory-specific CLAUDE.md does.

Re: Harness engineering for self-improvement

#86

Post by shepherdjerred »

Drakim wrote: post #9
Kinrany wrote: post #8

The quest for Torment Nexus continues

If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first.

https://theonion.com/sam-altman-if-i-dont-end-the-world-some...

Re: Harness engineering for self-improvement

#87

Post by cindyllm »

AlexErrant wrote: post #83
jagenabler2 wrote: post #81

Earlier quoted context omitted.

I’m just holding out for when this stuff becomes mainstream and automatically built into tooling. I don’t want to hold a retro with my agent. I actually want to spend as little time as possible talking with it.

Sure, you can "vibe/auto-retro". But in my experience the retro often results in "higher-order" structural changes. In starker terms: if you're ok with "vibe principles", "vibe invariants", "vibe lint-rules", "vibe skill updates", and "vibe agent-tooling"... well. That's your call. Thankfully after a dozen or so retros the low hanging fruit starts to disappear. But until then, you've no idea what's there.

[dead]

Re: Harness engineering for self-improvement

#88

Post by aaronzhang42 »

megadragon9 wrote: post #82

I did something similar. It started off as a "self-improving agent" project, inspired by autoresearch, then later on I reframed it as "harness training" (discrete program search) borrowing the mental model from ML training. I "trained" the harness on a subset of Terminal-Bench 2.0 tasks while keeping the LLM (local Qwen3.6-35B A3B) frozen. Making LLM inference and the task environments fully deterministic was necessa…

[dead]

Re: Harness engineering for self-improvement

#89

Post by aaronzhang42 »

bisonbear wrote: post #48
sulam wrote: post #46

Earlier quoted context omitted.

You've just described what the AI industry refers to as "evals", a collection of which forms a particular benchmark. I definitely encourage you to define your own evals, because the public benchmarks are often either saturated (largely solved and only going to see small improvements going forward) or seemingly not predictive of real world performance. I could also go further and speculate that they are in the trainin…

Thanks for the comment - I’ve actually been working on something to generate evals from private repos, as I 100% agree that public benchmarks are either contaminated (as OpenAI found with SWE-bench Verified), not high quality (as OpenAI found with SWE-bench Pro), or in the case of “good” benchmarks (Cognition’s Frontier Code comes to mind), non-representative of a specific repo’s work. Sharing some challenges / learn…

[dead]

Re: Harness engineering for self-improvement

#90

Post by tosh »

GrinningFool wrote: post #76
radlad wrote: post #59

Earlier quoted context omitted.

Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable? As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible. My CLAUDE.md, memories, and skills are all about either (a) adherence to project standa…

You don't have to remove them - you can move things that are not important to every prompt to aseparate small docs that are referenced in agents.md with ,"when needed, reference these files:" and list them in form * relpath - content summary" or similar. . When you want those instructions to be followed, mentiont them in your prompt. "Test this following procedures on docs/test.md". (I've found I don't have to do tha…

great point, progressive disclosure is a good pattern for context management
Post reply on HN