Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is a Harness?

earendil.com

Post reply on HN
Print view View original

71–80 of 200 posts

Re: What Is a Harness?

#71

Post by dpritchett »

pdhborges wrote: post #68
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

So you still have CLIs but they have I presume an help command that describes the capabilities right. Could you give an example of an accounting guardrail you created?

I’ve also found that Claude and friends are eerily good at using classic Unix CLI tools so I build mine in the same style, not unlike the `gh` CLI from GitHub, though with an agent-first design shape.

Usually I’m returning TSV as a default format and I add a `help-all` subcommand to list every available command at once when needed. Another thing that helps is adding just-in-time context-sensitive hints, such as: user has just run a list query with at least one result. Add a one-liner to the response explaining the command shape for getting the detail view of the first response.

In terms of skill files, I like to have my CLI generate them dynamically at runtime by walking their own current command tree and then feeding that through a text template.

Examples from a public project: https://github.com/radiusmethod/gitlab-kiosk/blob/main/skill...

Re: What Is a Harness?

#72

Post by newsomix9xl »

Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

Can you post a generic version of code for this somewhere (e.g. codeberg or whatever)?

I find your description intriguing but I'd like to see it to make sure I understand it.

Re: What Is a Harness?

#73

Post by epolanski »

theturtletalks wrote: post #3

Harnesses are the next frontier. If LLMs are electricity, harnesses are the “electronics.” Right now, it’s like an AC vs DC between Claude and ChatGPT, but once that settles, the harnesses will be the actual value providers. And Pi is the best harness because of the amazing extension system. You can build extensions that turn Pi into a stock trader, software factory, anything. I tried switching to another harness but…

Both Claude and codex are unappealing, crap, generic agents that you have 0 control over.

Don't understand what people see in them.

Re: What Is a Harness?

#74

Post by xrd »

Does anyone have a suggestion for a harness that is good at handoff?

When I say handoff, I mean:

  * handoff from a terminal CLI to webui (on a phone)? 
  * handoff from one team member, to another?
  * handoff from one communication modality, like writing a prompt in a TUI, to email? 
  * handoff from one model to another, or one provider (openrouter)( to another (llama.cpp)
Does such a thing exist?

I used to think that a PR would be a good place to centralize all this. Who cares what IDE, or developer, or location. But, now I feel like an agent harness might contain that better.

Why do I want handoff? I keep losing context of where my harness is running. Sometimes I am inside an isolated VM. Sometimes I'm on my laptop, sometimes I'm on my home machine with the big GPU for local models. If I could spin up a harness that could identify itself inside my tailscale network, then I could probably have a single web UI which allows me to keep all that context straight.

I'm tempted to experiment with Pi to configure such a thing. But, perhaps there are patterns out there already with a harness I have not considered.

Re: What Is a Harness?

#75

Post by layer8 »

_superposition_ wrote: post #50
mpawelski wrote: post #11

Earlier quoted context omitted.

> Harnesses are the next frontier. If LLMs are electricity, harnesses are the “electronics.” I really though this comment was a satire ...

In a sense they are the last frontier imo. At some point a harness will be built that can modify itself to fit the needs of the majority of people's workflows and evolve with them.

Then people will want to share and exchange their evolved harnesses. Ways will be found to modularize certain aspects to enable mixing and matching.

I’m thinking of how in cyberpunk, people are replacing their cybernetic enhancements all the time. You could alternatively bioengineer your own body towards the desired outcomes, but that’s more constrained by the trajectory your body has already taken, whereas the promise of cybernetic parts is that they are more independently replaceable. (Probably an illusion in practice, but I’m talking about the fictional ideal.)

As another analogy, monolithic software tends to quickly become hard to change significantly, whereas a plugin architecture tends to be more flexible and modular, and people can share and combine their various plugins.

Re: What Is a Harness?

#76

Post by theturtletalks »

grey-area wrote: post #69
theturtletalks wrote: post #43

Earlier quoted context omitted.

I didn’t come with the electricity idea, it was Sam Altman saying it will be like a utility down the line and metered[0]. What would the “electronics” be in your opinion? 0. https://www.businessinsider.com/sam-altman-ai-utility-electr...

Altman is a salesman selling flimflam to people who should know better. His idea of metering is predicated on the thing he’s selling being AGI, it is not, and all his predictions have turned to dust. Also that isn’t how metaphors work - they illuminate by comparison, if the comparison is not close they are not useful.

If it is metered and like a utility, Sam Altman will not benefit alone. All the models will have plateaued and you can swap for any of them. Then the only differentiator is the harness.

I don’t believe in AGI, but that doesn’t mean I don’t find AI useful. I just understand that the correct harness can take them to the next level.

Re: What Is a Harness?

#77

Post by jbstack »

grey-area wrote: post #25
mpawelski wrote: post #11

Earlier quoted context omitted.

> Harnesses are the next frontier. If LLMs are electricity, harnesses are the “electronics.” I really though this comment was a satire ...

Sadly, many people have bought into the cult that LLMs will lead to AGI. I guess if that is your worldview then all this babbling about new frontiers makes more sense. They probably used an LLM to come up with this bizarre metaphor.

I find it difficult to understand people who are wildly skeptical about LLMs leading to AGI (assuming we can even agree on what that means). Consider:

- They can already reason better than many humans and are still improving all the time

- Harnesses are improving all the time

- We're already exploring things like long term memory, long term goals, and other things that humans have which LLMs traditionally lack

- An AI agent can read and reason about every piece of AI research ever published, including looking for insights that humans may have missed. A team of humans could never do this even if they dedicated their whole lives to it.

- They can design and execute experiments on a mass scale to determine what does and doesn't work

- Large AI labs have more than sufficient resources and motivation to throw at the problem, and are in fact doing this.

Re: What Is a Harness?

#78

Post by freepiai »

I think a harness is kind of anything around the intelligence that allows the intelligence to be applied towards something, some sort of task. A great (if off-color) example I remember hearing was how Steven Hawking was brilliant, but really needed that computer setup to be able to apply his intelligence. It really stands out to me as such a clear visual example of what a harness actually is.

Anyway I've been building my own harness on top of pi- www.freepi.ai (it's based on Pi, but now I have an OpenAI compatible endpoint so I'm thinking of it more like free-api :-) ). Basically ad+training supported so I can offer completely free inference. It's really important to me that we don't have harnesses and intelligence trapped in a "have and have not" world. If we don't all have access to intelligence we will end up in a dark place.

Thats again where the visual of Steven Hawking and the wheelchair really stand out in my mind. It's not enough to have the raw intelligence, we need a really good wheelchair too.

Re: What Is a Harness?

#79

Post by Syntaf »

pdhborges wrote: post #68
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

So you still have CLIs but they have I presume an help command that describes the capabilities right. Could you give an example of an accounting guardrail you created?

Yeah the CLI can provide schema for commands via the usual ‘—help’ syntax, so agents are able to discover + explore commands on their own.

As for an example: if our agent wants to book a journal entry to cash accounts for a client, it MUST provide receipt and directly link the transaction from the clients bank feed, if it attempts to do so without the requisite information we deny the tool call and ask the agent to escalate back to the client for proof of receipt.

Often times this results in the agent not doing the work and instead sending a message back to the client asking for proof of the transaction.

For humans on our platform there may be valid situations where we’d want to allow this, but for our agent this is a hard guardrail thus why it’s not just standard validation for any JE posting on our platform.

Post reply on HN