Harness Engineering
github.com
Harness Engineering
1–10 of 39 posts
Re: Harness Engineering
#2Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.
Re: Harness Engineering
#3The mother of all prompt injections.
Do others at OpenAI use this?
Re: Harness Engineering
#4[dead]
Re: Harness Engineering
#5Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.
This is interesting. I'd already had a conversation with my harness ( pi ) about incorporating continuous improvement. This is a great deal better than what I came up with.
Re: Harness Engineering
#6Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.
One challenge/opportunity I've had is harnessing really wide running cheap agents. Any thoughts on how to move really cheap agents beyond basic summarization so we can go broader than the pricing of frontier llms allows?
Re: Harness Engineering
#7Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.
This is interesting. I'd already had a conversation with my harness ( pi ) about incorporating continuous improvement. This is a great deal better than what I came up with.
Glad to hear it! Good luck, have fun. The agents tend to do a pretty good job incorporating these ideas. This was an unexpected thing we learned when publishing the initial harness engineering post.
Re: Harness Engineering
#8Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.
One challenge/opportunity I've had is harnessing really wide running cheap agents. Any thoughts on how to move really cheap agents beyond basic summarization so we can go broader than the pricing of frontier llms allows?
Your “really cheap” agents can’t be so cheap that they do not have good tool calling skills. But! Using bigger models to put guardrails in place as static verifiers allows lower complexity changes to “self steer” as tests fail, which means coming down on the cost curve is more effective.
Re: Harness Engineering
#9Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.
How do you view harness engineering as an organic development that emerges from its use within a specific domain? Basically the meta-loop that allows an agent to tailor its harness to improve outcomes based on performance feedback. I use Pi a lot and I'm very interested in "self-assembling software".
One concrete example might be maintaining a conventions document per-project that covers how to name things semantically from a list of nouns and verbs. The idea is that LLMs are often not very globally aware, but it's important to maintain coherence across a code base in order for it to scale (in size and over time). Sometimes an LLM might call the same concept a Materialization, sometimes a Projection, and its not useful if its using two terms interchangeably without purpose.
Basically, how are you maintaining coherence when there isn't a human steering the code beyond providing requirements and validation directives?
I see you have relevant context in the repo like https://github.com/lopopolo/harness-engineering/tree/trunk/d... but I'm curious what exists beyond context. Do you use any tooling to steer this type of thing more consistently?
Re: Harness Engineering
#10Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.
How do you view harness engineering as an organic development that emerges from its use within a specific domain? Basically the meta-loop that allows an agent to tailor its harness to improve outcomes based on performance feedback. I use Pi a lot and I'm very interested in "self-assembling software". One concrete example might be maintaining a conventions document per-project that covers how to name things semantical…
Your example is super amenable to vibing some tests. As an example, I’ve been able to ban `number` from representing a duration by walking the AST in a linter to fail if var or param names that look like the end in millis or ms or sec appear. This is largely good enough. If you see that “drifting” behavior appear more than once, you have enough to stop and force the agent to write some static verifiers that reject all but the option you want. For a closer example, we did this with zod schemes and their corresponding inferred types to be universally ZPascalCase and PascalCase instead of camelCaseSchema and CamelCase