Thinking that harnesses and agents can improve without human involvement, even if it works, will yield a much lower growth rate than if a human gets involved in the loop.
Harness engineering for self-improvement
61–70 of 96 posts
Re: Harness engineering for self-improvement
#62Very enjoyable article. Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model. What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid developme…
I don't think so. A basic consumer-oriented harness is a commodity (zero moat: you can ask one harness to write you another).
State-of-the-art models have more of the workflow sensibilities built in, and don't need as much help from the harness.
Where harness helps the most is very customised personal workflows (not a textbox for a prompt, but an end-to-end IDE for how you develop software) But then it's best to have your own, rather than some closed-source rigid product.
Harnesses help smaller models, and very tightly hand-holding harnesses are needed for models runnable locally. I think that is very valuable to users, but avoiding paying for the biggest models is the opposite of what the frontier labs want.
Re: Harness engineering for self-improvement
#63I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…
> The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.
I've built and my team uses plannotator [1] for specs and code review. It stores all human annotated feedback, which we compile from time-to-time and iterate on coding standard skills for automated review (that seems to get better with the more feedback we provide).
You could write a script that does the same for your PR review comments, which it sounds like you maybe do with a bit more structure.
Re: Harness engineering for self-improvement
#64Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp). If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools. The article is a valuable resource, thanks to the author.
I'm curious to learn more about your experiences here, what you've learned, if you've enjoyed using the REPL with agents
Re: Harness engineering for self-improvement
#65Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp). If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools. The article is a valuable resource, thanks to the author.
The dream of the AI lisp machine ! Did you build in the functionality that the agent can on the fly rewrite its own (harness)-code?
I stuck a lisp interpreter inside a web server and its a pretty cool pattern. I do have a basic harness but I'm still kind of just using the REPL from an external agent over REST, which seems to work just as well without having to go through the brain damage of building a harness
Re: Harness engineering for self-improvement
#66in america AI harnesses you.
Re: Harness engineering for self-improvement
#67[dead]
Re: Harness engineering for self-improvement
#68IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations. I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors…
What date was the peak? If it is today it’s not something you can know so I assume you think the peak was many months ago.
I think it is now or close to now - and not because there is no more to be gained - but rather because it is now clear that you can gain more with the right agent environment.
But benchmark saturation is also something to account for.
Re: Harness engineering for self-improvement
#69We've using auto-research for harnesses and it's surprisingly powerful. It's amazing how many problems are easy to spot and fix from traces.
Few key things were required to get it working well: 1) let it read a heap of prod traces to spot real issues. 2) let it write it's own tools (example "loading context" goes from 20k tokens across 15 tool calls to 800 tokens and 1 call to session_context tool). 3) you need evals and val/test splits, it will reward hack. 4) you need proper tooling (synthetic users, synthetic tools) for it to be able to crunch for 12 hours and produce something. 5) you need the optimization target to be a reasonable size: not your 1M line codebase, but a lighter agent harness (pi harness, skill only, Kiln harness).
Re: Harness engineering for self-improvement
#70[deleted]