Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harness engineering for self-improvement

lilianweng.github.io

Post reply on HN
Print view View original

31–40 of 96 posts

Re: Harness engineering for self-improvement

#31

Post by intrasight »

sbysb wrote: post #28
kriro wrote: post #27

Very enjoyable article. Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model. What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid developme…

There is a really good video by the author of pi.dev (which I have used to build some of my own harnesses): https://www.youtube.com/watch?v=RjfbvDXpFls The basis of the argument is that the labs are constantly pushing updates to their system prompts that are used in claude code or codex, which are exceptionally bloated and change the sand beneath your feet with every update. By rolling your own harness, as long as yo…

My own opinion is that the "harness companies" (Anthropic, OpenAI, Cursor etc.) are fucked long-term since customers will tire of that quicksand. As the models improve, we will tire of that harness crap.

Re: Harness engineering for self-improvement

#32

Post by anon373839 »

kriro wrote: post #27

Very enjoyable article. Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model. What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid developme…

> Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts?

They're trying. I see a few main avenues:

1. Fitting the models to their specific harness, so that if you want peak model performance, you're stuck with their harness. But this only works if there aren't alternative models that are similar in capability or good enough that don't have that restriction.

2. Locking you out of the harness so that you eventually are just letting it do "stuff" with your data and system, although you don't get to see what the stuff is or why it's doing it. This is the pattern discussed in that Earendil blog post, "The session you cannot take with you." Encrypted reasoning tokens, secret agent prompts, perhaps eventually not even seeing what files are being read or what data is being sent back to their server. This way, you are also shackled to their harness because nothing is portable. But it only works if you trust them implicitly and don't have alternative models and harnesses that don't treat you this way.

3. Tying subscription pricing to the use of their harness, so that it's financially punitive to use another one. This is what Anthropic does. But again, it only works if there aren't alternative models and harnesses that work similarly well for you and don't have that restriction.

4. Marketing. Anthropic is leaning into this one heavy, from what I can tell, based on the constant ads I see for Claude Code. Can it work? People do overpay for things like vodka in fancy bottles that are chemically identical to the cheap stuff. Is Veblen-good AI a trillion dollar business, though?

I think the pressure from open source innovation in models and model tooling is going to make it a tough row to hoe. But I'm biased, as I'm actively rooting for Openthropic's demise.

Re: Harness engineering for self-improvement

#33

Post by lmf4lol »

mark_l_watson wrote: post #22

Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp). If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools. The article is a valuable resource, thanks to the author.

The dream of the AI lisp machine ! Did you build in the functionality that the agent can on the fly rewrite its own (harness)-code?

Re: Harness engineering for self-improvement

#34

Post by zby »

IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations.

I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors. Some day the llm prose will improve.

I have even a preliminary theory on what is needed for the positive feedback loop: https://zby.github.io/commonplace/articles/reflective-self-i... - (this is not stable yet - but I think you can give it to your agent to read :).

Re: Harness engineering for self-improvement

#35

Post by keremturhan »

bob1029 wrote: post #25

I think the biggest problem with RSI is the tendency of models to try and find "clever" solutions when presented with very hard problems (aka cheat). I just got done asking gpt5.5 to make some suggestions that would improve the convergence properties of a symbolic ML experiment and the very first thing it did was add an instruction for directly emitting bytes and a way to store those verbatim in the model. Perfect sc…

[flagged]

Re: Harness engineering for self-improvement

#36

Post by erwincoumans »

Thanks for the really nice in-depth post! Hoping for a future one about:

"Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post."

Re: Harness engineering for self-improvement

#37

Post by robbru »

cahaya wrote: post #21

Thanks for the post. For https://Document.bot (Kinda Cursor IDE for knowledge workers), im already trying to improve the harness (besides spotting bugs) using hillclimb experiments. More and more i'm using a AI harness engineering skill in Codex to further improve the app. This blog post helped me to improve the skill a bit.

Very interesting, will give it a demo.

Re: Harness engineering for self-improvement

#39

Post by mark_l_watson »

lmf4lol wrote: post #33
mark_l_watson wrote: post #22

Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp). If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools. The article is a valuable resource, thanks to the author.

The dream of the AI lisp machine ! Did you build in the functionality that the agent can on the fly rewrite its own (harness)-code?

No, but a good idea. I want something simple, practical, understandable. I think I will write a short book on this general topic.

Re: Harness engineering for self-improvement

#40

Post by ianm218 »

zby wrote: post #34

IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations. I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors…

Is there any reason to think that training weights has peaked rather than is accelerating? It feels like now they are increasingly able to pick some low hanging fruit by using the models in order to improve themselves and test optimizations.
Post reply on HN