Harness engineering for self-improvement • Hacker News Board

Re: Harness engineering for self-improvement

#51

Post by tosh » Tue, Aug 04, 2026, 5:14 PM UTC

bisonbear wrote: post #41

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…

maybe a bit counter-intuitive but:

I found that removing

  - system prompt
  - skills
  - agents.md
  - mcps

+ reducing tools to just 1 (sh)

gives better results than having 'more' of them

(e.g. look at these traces to see more vs less in action:)

https://smolenv.com/t/nested-template-includes-60636/

not saying the right context does not help

(it definitely does!, but it's not trivial to provide the right context)

Re: Harness engineering for self-improvement

#52

Post by [deleted] » Tue, Aug 04, 2026, 5:16 PM UTC

bisonbear wrote: post #41

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…

[deleted]

Re: Harness engineering for self-improvement

#53

Post by radlad » Tue, Aug 04, 2026, 5:18 PM UTC

tosh wrote: post #51

bisonbear wrote: post #41

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…

maybe a bit counter-intuitive but: I found that removing - system prompt - skills - agents.md - mcps + reducing tools to just 1 (sh) gives better results than having 'more' of them (e.g. look at these traces to see more vs less in action:) https://smolenv.com/t/nested-template-includes-60636/ not saying the right context does not help (it definitely does!, but it's not trivial to provide the right context)

'better results' in terms of what though? A benchmark, or code that I would actually click "approve" on in a pull request scenario?

Re: Harness engineering for self-improvement

#54

Post by tosh » Tue, Aug 04, 2026, 5:21 PM UTC

radlad wrote: post #53

tosh wrote: post #51

Earlier quoted context omitted.

maybe a bit counter-intuitive but: I found that removing - system prompt - skills - agents.md - mcps + reducing tools to just 1 (sh) gives better results than having 'more' of them (e.g. look at these traces to see more vs less in action:) https://smolenv.com/t/nested-template-includes-60636/ not saying the right context does not help (it definitely does!, but it's not trivial to provide the right context)

'better results' in terms of what though? A benchmark, or code that I would actually click "approve" on in a pull request scenario?

apologies, I should have clarified the 'better' claim

  - same task result (passed)
  - finished faster
  - fewer tokens, less cost
  - fewer requests for inference
  - fewer tool calls
  - less peak RAM

Re: Harness engineering for self-improvement

#55

Post by nowittyusername » Tue, Aug 04, 2026, 5:24 PM UTC

bisonbear wrote: post #41

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…

Ive been building my own ai voice agent harness from scratch for close to a year now and following good software architecture practices is a good start. So those rules have to be coded in agents.md somewhere also really helps to have a "vision" section or "spirit of the project" section that describes what the end goal vaguely looks like and things I care about in achieving for the project. This prevents agent from being brittle and "single minded" about its work. But yeah vibes most of the time is how I've also been doing it, but I did find one very important thing that has really sped up my work. So I figure I'd share it here. And that advise is to ignore front end design at all costs until the very end and you are ready to launch. UI related woes kill any type of inertia and are responsible for most of the issues. So I decided abandon all UI stuff until the very end and just focus on iterative refinement, cutting, and other back-end related work and its been great ever since. Ideas can be had, tested, validated/invalidates and you ' get stuck on the optimization thought loop. Recently I have started to consider how i can fully automate the development process as the capabilities are there but designing the instructions on how to do this well and how to handle niche cases without getting me involved takes careful planning in writing out the guide so that will be interesting to see once i get there.

Re: Harness engineering for self-improvement

#56

Post by AlexErrant » Tue, Aug 04, 2026, 5:49 PM UTC

bisonbear wrote: post #41

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…

I literally hold a "retro" with my agent after a session.

> Let's conclude with a retro. Did you run into any issues during this session that you think could be improved? Any failed tool calls, confusing docs/prompts, or tricky wording that took you effort to figure out, etc? Any final thoughts that you want to raise? Anything minor you didn't mention? Help make this codebase easier for the next agent to work in.

It has ideas for coding changes, spots unmentioned small bugs, suggests invariants (I have a high-level list of invariants to keep), principles to adopt, lint rules, tooling tweaks (I've built SO MUCH agent-tooling; so many useful bash scripts), skill-file updates, follow up work, all kinds of stuff.

Listen to your agent's whinging.

Re: Harness engineering for self-improvement

#57

Post by victorbjorklund » Tue, Aug 04, 2026, 5:51 PM UTC

zby wrote: post #34

IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations. I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors…

What date was the peak? If it is today it’s not something you can know so I assume you think the peak was many months ago.

Re: Harness engineering for self-improvement

#58

Post by bisonbear » Tue, Aug 04, 2026, 5:55 PM UTC

tosh wrote: post #51

bisonbear wrote: post #41

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…

maybe a bit counter-intuitive but: I found that removing - system prompt - skills - agents.md - mcps + reducing tools to just 1 (sh) gives better results than having 'more' of them (e.g. look at these traces to see more vs less in action:) https://smolenv.com/t/nested-template-includes-60636/ not saying the right context does not help (it definitely does!, but it's not trivial to provide the right context)

This is the right direction (Anthropic removing 80% of the system prompt https://claude.com/blog/the-new-rules-of-context-engineering...)

However what I wonder how we make these decisions on what to remove. Certainly some subset of the skills / AGENTS.md / prompt / tools do actually help agent performance, the problem is in identifying those, throwing away the rest, and validating that this didn't actually hurt the agent performance

Re: Harness engineering for self-improvement

#59

Post by radlad » Tue, Aug 04, 2026, 5:57 PM UTC

tosh wrote: post #54

radlad wrote: post #53

Earlier quoted context omitted.

'better results' in terms of what though? A benchmark, or code that I would actually click "approve" on in a pull request scenario?

apologies, I should have clarified the 'better' claim - same task result (passed) - finished faster - fewer tokens, less cost - fewer requests for inference - fewer tool calls - less peak RAM

Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable?

As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible.

My CLAUDE.md, memories, and skills are all about either (a) adherence to project standards and guidelines, (b) product decisions which impact future code, and (c) instructions on how to prototype and work in my environment.

Removing these instructions would mean more turns with the AI to get the desired result.

Re: Harness engineering for self-improvement

#60

Post by tosh » Tue, Aug 04, 2026, 6:00 PM UTC

radlad wrote: post #59

tosh wrote: post #54

Earlier quoted context omitted.

apologies, I should have clarified the 'better' claim - same task result (passed) - finished faster - fewer tokens, less cost - fewer requests for inference - fewer tool calls - less peak RAM

Yes, my question is directed around how you are determining "task result." Is this measuring whether the code works, or whether it is maintainable? As with human-powered coding, we read code far more frequently than we write it. It's worth spending a little extra (time|tokens) during authorship to make future maintenance feasible. My CLAUDE.md, memories, and skills are all about either (a) adherence to project standa…

in this case there was a hidden grader that checked if the implementation was correct (because that was the easiest thing to check), all 3 agents cleared this hurdle in all 9 runs

I agree, next it makes sense to try more open ended tasks + have humans (and/or multiple models) grade the runs and their results