Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Harness Engineering

github.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: Harness Engineering

#11

Post by lopopolo »

lopopolo wrote: post #10
hankbond wrote: post #9

Earlier quoted context omitted.

How do you view harness engineering as an organic development that emerges from its use within a specific domain? Basically the meta-loop that allows an agent to tailor its harness to improve outcomes based on performance feedback. I use Pi a lot and I'm very interested in "self-assembling software". One concrete example might be maintaining a conventions document per-project that covers how to name things semantical…

Your example is super amenable to vibing some tests. As an example, I’ve been able to ban `number` from representing a duration by walking the AST in a linter to fail if var or param names that look like the end in millis or ms or sec appear. This is largely good enough. If you see that “drifting” behavior appear more than once, you have enough to stop and force the agent to write some static verifiers that reject al…

And to address your broader question, yes this is a form of RSI and to me a vastly superior approach to fine tuning since it allows adopting new model releases without throwing anything away while still having the same effect on improving adherence to local acceptance criteria.

Re: Harness Engineering

#13

Post by slopinthebag »

lopopolo wrote: post #2

Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.

What motivated you to quote your own quote (??) in your readme claiming to boost productivity by 100x, and where did you derive that number from?

"When a quote sounds profound enough, reality usually nods out of politeness, without echoes is just a sentence wearing pajamas." - slopinthebag

Re: Harness Engineering

#15

Post by hahahaa »

lopopolo wrote: post #2

Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.

Thanks. I am guessing you have to try stuff and build tacit experience. No other way, just get stuck in and try stuff, then try and learn bits from others?

Re: Harness Engineering

#16

Post by lopopolo »

hahahaa wrote: post #15
lopopolo wrote: post #2

Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.

Thanks. I am guessing you have to try stuff and build tacit experience. No other way, just get stuck in and try stuff, then try and learn bits from others?

Basically yea. It is the only way to learn how to outrun your priors on what “high ambition” looks like. The labor that goes into implementation is an uncapped resource now.

Re: Harness Engineering

#17

Post by mcapodici »

lopopolo wrote: post #8
mips_avatar wrote: post #6

Earlier quoted context omitted.

One challenge/opportunity I've had is harnessing really wide running cheap agents. Any thoughts on how to move really cheap agents beyond basic summarization so we can go broader than the pricing of frontier llms allows?

Your “really cheap” agents can’t be so cheap that they do not have good tool calling skills. But! Using bigger models to put guardrails in place as static verifiers allows lower complexity changes to “self steer” as tests fail, which means coming down on the cost curve is more effective.

I was experimenting and found deepseek-v4-flash and found it cheap (way cheap compared to sota models) and perfectly good at tool calling. I did a post on it https://martincapodici.com/2026/07/18/weekly-ai-learnings-3/

So $1.5 for 40m tokens I guess would cost much more with sota (but would need less tokens perhaps).

Re: Harness Engineering

#18

Post by kubb »

slopinthebag wrote: post #13
lopopolo wrote: post #2

Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.

What motivated you to quote your own quote (??) in your readme claiming to boost productivity by 100x, and where did you derive that number from? "When a quote sounds profound enough, reality usually nods out of politeness, without echoes is just a sentence wearing pajamas." - slopinthebag

I would also love to know. Sounds dubious to say the least.

Re: Harness Engineering

#19

Post by lopopolo »

hahahaa wrote: post #15
lopopolo wrote: post #2

Hi folks, author here and also author of the seminal OpenAI blog on this topic. Let me know how I can help you all let it rip.

Thanks. I am guessing you have to try stuff and build tacit experience. No other way, just get stuck in and try stuff, then try and learn bits from others?

The models are very good now so the feedback cycle on these meta adjustments is much tighter. Yesterday I was able to one shot a Liquid Glass, HIG-compliant and localized DICOM image viewer (frame by frame and looping video) with Apple Intelligence for de-jargoning the series details. Took 30 minutes. But the app had 60% CPU because it was not caching the decoded JPEGs. I can do a point in time fix for that of course, but the more interesting thing is why that misaligned code was permitted to be generated in a “done” artifact in the first place. What other misaligned code from a perf perspective might there be? And how do I intervene into the system that produced this software to make these misalignments statically not meet acceptance criteria?

Re: Harness Engineering

#20

Post by ricardobeat »

What percentage of this was written with AI?

> A command can feel ambient without making its credential model-visible.

I'd be wary of following guidelines on how to build human-computer interaction systems that weren't written with full human oversight. This kind of wording makes me wonder if these are actual recommendations or just the AI pattern-matching and hallucinating something that looks coherent.

Post reply on HN