Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harness engineering for self-improvement

lilianweng.github.io

Post reply on HN
Print view View original

91–96 of 96 posts

Re: Harness engineering for self-improvement

#91

Post by GrinningFool »

radlad wrote: post #84
GrinningFool wrote: post #76

Earlier quoted context omitted.

You don't have to remove them - you can move things that are not important to every prompt to aseparate small docs that are referenced in agents.md with ,"when needed, reference these files:" and list them in form * relpath - content summary" or similar. . When you want those instructions to be followed, mentiont them in your prompt. "Test this following procedures on docs/test.md". (I've found I don't have to do tha…

That's what MEMORY.md and subdirectory-specific CLAUDE.md does.

Aren't subdirectory specific Claude files layered in on top? Seems like more required content when the direction I'm suggesting is more like lazy-loading.

Re: Harness engineering for self-improvement

#92

Post by avadodin »

jtr1 wrote: post #50
bisonbear wrote: post #41

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever. The first step I see towards this is building a gener…

Just working on my own stuff so YMMV, but from a cost reduction standpoint, the two most outsized ROI wins came from making sure the agent runs all terminal commands in quiet and using a codebase index mcp to speed up understanding and reduce full file reads [0]. I've also found the latter also dramatically improved the quality of output. [0] https://github.com/DeusData/codebase-memory-mcp

We actually had a lot of the tooling and process rules we now give out to LLMs so that they do what we want them to do optimally figured out and theoretically in place way before LLMs.

Turns out everyone had just been vibe–managing people this whole time so there was still that guy burning salary "tokens" going through a million lines of on–demand grep output.

Re: Harness engineering for self-improvement

#93

Post by radlad »

GrinningFool wrote: post #91
radlad wrote: post #84

Earlier quoted context omitted.

That's what MEMORY.md and subdirectory-specific CLAUDE.md does.

Aren't subdirectory specific Claude files layered in on top? Seems like more required content when the direction I'm suggesting is more like lazy-loading.

They load only when operating on files in that directory. i.e. Lazy loading?

For example, I have one in my schema migrations directory that explains how to write migrations with appropriate concern for forwards- and backwards-compatibility in our deployment model.

Re: Harness engineering for self-improvement

#94

Post by GrinningFool »

radlad wrote: post #93
GrinningFool wrote: post #91

Earlier quoted context omitted.

Aren't subdirectory specific Claude files layered in on top? Seems like more required content when the direction I'm suggesting is more like lazy-loading.

They load only when operating on files in that directory. i.e. Lazy loading? For example, I have one in my schema migrations directory that explains how to write migrations with appropriate concern for forwards- and backwards-compatibility in our deployment model.

This type of content I use this for is not necessarily directory-specific - git operations as a simple example. I have workflows that are only relevant in specific situatoins, but they apply to the whole repository.

This is effective only if you're guiding it heavily though - this is a very human-supervised workflow, and I'm telling it when it's time to take specific actions. It will read the appropriate instructions file when the action is first referenced, but not until then.

Re: Harness engineering for self-improvement

#95

Post by FacelessAICoder »

thanks for posting the question - this space certainly keeps evolving

I've been trying to tackle various aspects of traceability and validation at scale... embedding the self improvement & validation in various iterations of Ralph Loops.... here's a deeper write up: https://dataspheres.ai/pages/dataspheres-ai/spec-driven-deve...

The tools & methodologies keep changing everyday - so I mostly just iterate on my own tooling https://github.com/geekdreamzz/ari-dai-skills so I can keep learning and iterating.... it's on my todo list to create some videos demoing it...

At a high-level - I find state management is really hard at scale in addition to managing all the context for specific situations. What I do is create a graph that tracks your original.... prompts specs tasks ai-generated code/content.... so it's never a question on why/what since it's all instrumented into a chain. The tasks have clear validation criteria and gates between statuses to adversarially challenge the status updates and validation. Where appropriate I have playwright take screenshots and review it as part of the chain. All of this gets tracked into a dashboard so it can just keep running while you're on the go..

I'm talking high-level and a bit all over the place because there is a lot of moving parts in this space. Would love to connect more on how you and others are tackling this. I'm using Claude Code to run the ari-dai-skills repo I linked above for all my initiatives now. I'm spending a lot of $$$ on Claude and more & more I'm itching to just invest in local compute and run more local LLM tasks autonomously because we're all iterating so much. I've been able to build a lot though but I think the open source models and the local inference is getting better.... so a new wave of tooling is on the horizon to make traceability and validation better so keep hacking ... best of luck... these harnesses are like a casino & sometimes we do win big ...I think we're a long way out from real consistency since there's just so many perspectives a harness facilitates.

Re: Harness engineering for self-improvement

#96

Post by scosman »

lerchmo wrote: post #71
scosman wrote: post #69

We've using auto-research for harnesses and it's surprisingly powerful. It's amazing how many problems are easy to spot and fix from traces. Few key things were required to get it working well: 1) let it read a heap of prod traces to spot real issues. 2) let it write it's own tools (example "loading context" goes from 20k tokens across 15 tool calls to 800 tokens and 1 call to session_context tool). 3) you need evals…

What is the fitness function , eval?

many evals, but yes
Post reply on HN