Thanks for the post. For https://Document.bot (Kinda Cursor IDE for knowledge workers), im already trying to improve the harness (besides spotting bugs) using hillclimb experiments. More and more i'm using a AI harness engineering skill in Codex to further improve the app. This blog post helped me to improve the skill a bit.
Harness engineering for self-improvement
21–30 of 96 posts
Re: Harness engineering for self-improvement
#22Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp).
If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools.
The article is a valuable resource, thanks to the author.
Re: Harness engineering for self-improvement
#23Earlier quoted context omitted.
If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first.
Since we all know that at this point, someone is definitely going to build the torment nexus, we all may as well get involved and help, so that at least we can exert some limited degree of control over the eventual character of the torment nexus. At least that way maybe we can make it a little less terrible. At any rate, it can’t make it worse than it would have been if we -didn’t- help build it!
whats the other option, touch grass? not likely!
Re: Harness engineering for self-improvement
#24Re: Harness engineering for self-improvement
#25I think the biggest problem with RSI is the tendency of models to try and find "clever" solutions when presented with very hard problems (aka cheat).
I just got done asking gpt5.5 to make some suggestions that would improve the convergence properties of a symbolic ML experiment and the very first thing it did was add an instruction for directly emitting bytes and a way to store those verbatim in the model. Perfect score achieved in record time but the result is pointless.
If you know the direction of improvement you need to head in, I think the current models could take you there. However, I do not think they are capable of deciding which direction is best, especially not when given some scalar quantity to min/max and autonomy over that process.
Re: Harness engineering for self-improvement
#26The quest for Torment Nexus continues
If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first.
If we don't do our best to awaken Dead Cthulhu, when He invariably awakens He will know that we didn't do our best and we'll be eaten last by the Great Old Ones upon Their return!
Re: Harness engineering for self-improvement
#27Very enjoyable article.
Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model.
What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid development of frontier models.
Re: Harness engineering for self-improvement
#28Very enjoyable article. Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model. What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid developme…
There is a really good video by the author of pi.dev (which I have used to build some of my own harnesses): https://www.youtube.com/watch?v=RjfbvDXpFls
The basis of the argument is that the labs are constantly pushing updates to their system prompts that are used in claude code or codex, which are exceptionally bloated and change the sand beneath your feet with every update.
By rolling your own harness, as long as you keep up with the latest advances and changes in the ecosystem, you capture a lot of the 'control' that LLM-based development feels like it strips from you.
Obvious disclaimer that I use pi.dev when I am aiming for consistency, not absolute quality. Custom harnesses on pi are what I ship, claude code is still my fallback if I need to make sure a PR is the highest absolute quality
Re: Harness engineering for self-improvement
#29Earlier quoted context omitted.
If we don't create the Torment Nexus first, somebody else who is much less responsible is gonna create the Torment Nexus before us. It's outright irresponsible to not take the lead, we might have to even give up on all safety concerns to make sure we make it to Torment Nexus IPO first.
Since we all know that at this point, someone is definitely going to build the torment nexus, we all may as well get involved and help, so that at least we can exert some limited degree of control over the eventual character of the torment nexus. At least that way maybe we can make it a little less terrible. At any rate, it can’t make it worse than it would have been if we -didn’t- help build it!
If you're going to get involved in a torment nexus project, at least do yourself a favor and research the different comp packages being offered and choose the best one.
And of course, a healthy skepticism of "the man" is essential when you're busting your ass building his unfathomably horrifying torture apparatus for him 40-50 hours a week (excluding PTO days and parental leave of course).
Re: Harness engineering for self-improvement
#30Fascinating article, thank you!