Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is a Harness?

earendil.com

Post reply on HN
Print view View original

121–130 of 200 posts

Re: What Is a Harness?

#121

Post by grim_io »

jbstack wrote: post #111
grim_io wrote: post #45

Earlier quoted context omitted.

I don't think so. What I can see is a world where we end up with a Chromium-shaped harness, a fully featured standard implementation everyone builds against, because doing every single thing yourself would be crazy. The antithesis to Pi, if you will.

I disagree with this. Unlike training models (which requires huge compute), harness development is available to anyone with an editor and ideas. That means that solo devs and small startups can still make meaningful progress. Also, having only a "standard implementation" makes no sense for a harness. A standard implementation would need to try to be as good as possible at all things. But you'd often want a specialise…

I don't see us tinkering with Pi in 5 years.

Some standard solution will emerge, which will be amplified by models being trained specifically to work with it.

Re: What Is a Harness?

#122

Post by msdz »

YZF wrote: post #92
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

> When I use LLMs I leave things as open as possible. I just give them the information they need and my ask.

How do you handle security?

Both “internally” against e.g. data loss, I’m assuming via limiting the harness, and “externally”, i.e. stuff like prompt injection risks?

Re: What Is a Harness?

#123

Post by rush86999 »

Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

I think you've really hit the mark on how the harness should be structured:

1. Guardrails - deterministic, social intelligence, team alignment & accountability 2. Learn by doing 3. make it stupid easy for the agent to research and access data 4. DRY

Research supports this. Try picking up some ideas from my harness: https://github.com/rush86999/atom

Re: What Is a Harness?

#124

Post by wonnage »

hombre_fatal wrote: post #100
wonnage wrote: post #95

Earlier quoted context omitted.

That works for well trod paths, e.g “fix ci” works exceedingly well. “why app slow” obviously doesn’t work because the task is underspecified. But in order to properly specify you either need an experienced engineer who knows how to narrow the problem domain, or you have to provide some template instructions/output formats (e.g, skills) which will invariably never fit the problem perfectly

I wouldn't agree. Sota models can do self-directed sampling, profiling, benchmarking, read call trees, etc. to give you a report of the app's bottlenecks and then recommend solutions that can be vetted. I do this constantly. As the upstream comment points you, you don't need to specify. Sota models are that good. And by being overprescriptive you can accidentally shut off branches that they would've taken, downgradin…

In my experience if you’re at the point where you have something to sample then the hard part is already done.

In a perfect world everything is covered by distributed tracing and the problems are only in your application code and the agent just needs to find the data

In reality the data is often missing or misleading. “Your observability sucks”? Yeah, but that’s life

Re: What Is a Harness?

#125

Post by cortesoft »

wonnage wrote: post #124
hombre_fatal wrote: post #100

Earlier quoted context omitted.

I wouldn't agree. Sota models can do self-directed sampling, profiling, benchmarking, read call trees, etc. to give you a report of the app's bottlenecks and then recommend solutions that can be vetted. I do this constantly. As the upstream comment points you, you don't need to specify. Sota models are that good. And by being overprescriptive you can accidentally shut off branches that they would've taken, downgradin…

In my experience if you’re at the point where you have something to sample then the hard part is already done. In a perfect world everything is covered by distributed tracing and the problems are only in your application code and the agent just needs to find the data In reality the data is often missing or misleading. “Your observability sucks”? Yeah, but that’s life

> “Your observability sucks”? Yeah, but that’s life

You could start by asking your AI "help me add better observability to our stack"

Re: What Is a Harness?

#126

Post by Aeolun »

timbowhite wrote: post #9
theturtletalks wrote: post #3

Harnesses are the next frontier. If LLMs are electricity, harnesses are the “electronics.” Right now, it’s like an AC vs DC between Claude and ChatGPT, but once that settles, the harnesses will be the actual value providers. And Pi is the best harness because of the amazing extension system. You can build extensions that turn Pi into a stock trader, software factory, anything. I tried switching to another harness but…

Pi's most popular extensions, by download count: https://pi.dev/packages?type=extension

Kinda ironic that the most popular extensions turn pi back into claude.

Re: What Is a Harness?

#127

Post by phs318u »

My 2c. A lot of what’s being discussed here may boil down to experience (or lack thereof) of writing good functional (and non-functional) requirements. Some developers are always thinking about implementation (or at least, implementation approach), and as a result they are very technically prescriptive. There’s nothing wrong with that per se if you’re actively driving towards a specific architecture (for your own reasons - which could be specified as non-functional reqs).

What others are describing is the BA skill of capturing good _business_ requirements ie. specifying outcomes and qualities rather than methods.

We should be clear about who’s being the BA, the architect and the developer here. At one end, a human vibe coder is acting as BA and the AI as architect, designer and engineer. At the other end, the human is a senior dev closely instructing a AI junior dev. All approaches are valid depending on what you need and the AI tools available (eg. SOTA model vs small self-hosted open-weights model).

Re: What Is a Harness?

#128

Post by avadodin »

YZF wrote: post #92
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

This vibe people sentiment is not wrong per se.

If you want outlier performance from these models it is best to just ask in the most high level prompt of the most minimal harness and let them loose.

Any extra information reduces their performance.

However, as often as these models output masterpieces, they also produce utter garbage so our current choice is for them to have a process to follow that can be reviewed by humans and LLMs.

Re: What Is a Harness?

#130

Post by speed_spread »

xrd wrote: post #74

Does anyone have a suggestion for a harness that is good at handoff? When I say handoff, I mean: * handoff from a terminal CLI to webui (on a phone)? * handoff from one team member, to another? * handoff from one communication modality, like writing a prompt in a TUI, to email? * handoff from one model to another, or one provider (openrouter)( to another (llama.cpp) Does such a thing exist? I used to think that a PR…

Hermes has /handoff to go from cli or desktop to IM, possibly other ways.
Post reply on HN