Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is a Harness?

earendil.com

Post reply on HN
Print view View original

181–190 of 200 posts

Re: What Is a Harness?

#181

Post by gf000 »

xrd wrote: post #74

Does anyone have a suggestion for a harness that is good at handoff? When I say handoff, I mean: * handoff from a terminal CLI to webui (on a phone)? * handoff from one team member, to another? * handoff from one communication modality, like writing a prompt in a TUI, to email? * handoff from one model to another, or one provider (openrouter)( to another (llama.cpp) Does such a thing exist? I used to think that a PR…

Not really solving all your cases, but I found tmux (or herdr) on a home server works pretty well. I just ssh into my home server and continue where I left off with the same claude/opencode open.

On a longer term, I think "assistant" style harnesses might help here, like vellum.ai. I no longer use that, but I asked it to create an ACP proxy through iroh (basically tailscale but on the application layer), and it managed to control claude on another device of mine. A friend did similar stuff with tailscale.

I have started writing a hobby harness with a web interface where I would like to support this "ACP proxy" mode natively, and also to make the models aware of different devices in some way and "move" work between them.

Re: What Is a Harness?

#182

Post by gf000 »

itishappy wrote: post #96
xrd wrote: post #74

Does anyone have a suggestion for a harness that is good at handoff? When I say handoff, I mean: * handoff from a terminal CLI to webui (on a phone)? * handoff from one team member, to another? * handoff from one communication modality, like writing a prompt in a TUI, to email? * handoff from one model to another, or one provider (openrouter)( to another (llama.cpp) Does such a thing exist? I used to think that a PR…

The session is "just" the raw chat history in it's entirety (human and agent) and can be disseminated as such. This is what enables swapping between models, you simply send the whole context. Not sure how others do it, but opencode stores sessions in a sqlite db and you can extract them and share them as needed. https://opencode.ai/docs/cli/#export Pro-tip: Building your own extremely minimal harness takes about 15m…

Another common format is `jsonl`, basically a json object per line containing a message

Re: What Is a Harness?

#183

Post by theptip »

stymaar wrote: post #164
YZF wrote: post #92

Earlier quoted context omitted.

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

The problem is that even Fable still make trivial yet high impact mistake when let on their own, and then you'd need to read the whole code to catch them… Meanwhile they are very good at implementating an explicit algorithm that you feed it to them.

The trick is to set up the harness so that the solution is easy to verify - you’ve profited as long as verification is cheaper than building, but ideally verification is close to automatic (not always achievable of course).

Generally you want to include objective/repeatable outputs as citations.

An example would be, if you invest in an awesome layered test rig (browser test, fuzz/property tests, very well reviewed unit/integration tests, etc.) then you should be able to add features by just reading the acceptance test and scanning unit tests.

Re: What Is a Harness?

#184

Post by theptip »

wonnage wrote: post #95
YZF wrote: post #92

Earlier quoted context omitted.

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

That works for well trod paths, e.g “fix ci” works exceedingly well. “why app slow” obviously doesn’t work because the task is underspecified. But in order to properly specify you either need an experienced engineer who knows how to narrow the problem domain, or you have to provide some template instructions/output formats (e.g, skills) which will invariably never fit the problem perfectly

Honestly I have had great success with “I’m worried here about cpu and latency, please rigorously profile and propose fixes”.

The models can build micro-benchmarks with a level of rigor that few could muster for a new feature.

I agree that if the issue is architectural they will struggle to understand that scope.

Re: What Is a Harness?

#185

Post by Syntaf »

floatrock wrote: post #179
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

Why a CLI over an MCP or even straight restful API with appropriate schema docs?

I don't have a strong argument for or against using MCP, it honestly comes down to familiarity.

In my own opinion, a CLI tool is going to be much more familiar ground for engineers -- I wouldn't expect the 200+ engineers at my company to all have read and understood the paradigms of the MCP protocol but I _would_ expect all of us to have a strong understanding of CLI tools and what a good/bad tool is.

Re: What Is a Harness?

#186

Post by flumes_whims_ »

floatrock wrote: post #179
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

Why a CLI over an MCP or even straight restful API with appropriate schema docs?

MCP clutters the context. Much better to have cli that has --help on subcommands so it can get the parts in needs in the current context.

Re: What Is a Harness?

#187

Post by wonnage »

pests wrote: post #134
wonnage wrote: post #95

Earlier quoted context omitted.

That works for well trod paths, e.g “fix ci” works exceedingly well. “why app slow” obviously doesn’t work because the task is underspecified. But in order to properly specify you either need an experienced engineer who knows how to narrow the problem domain, or you have to provide some template instructions/output formats (e.g, skills) which will invariably never fit the problem perfectly

> “why app slow” obviously doesn’t work because the task is underspecified Definitely not true and like everyone else is saying, shows how people still underestimate these models. I have been working on a simple vite + react app lately and commonly ask Gemini/Antigravity to just "improve speeds", "x is running slow, check it out" and have no complaints.

I’m not surprised it works on a simple app.

Re: What Is a Harness?

#188

Post by wonnage »

cortesoft wrote: post #125
wonnage wrote: post #124

Earlier quoted context omitted.

In my experience if you’re at the point where you have something to sample then the hard part is already done. In a perfect world everything is covered by distributed tracing and the problems are only in your application code and the agent just needs to find the data In reality the data is often missing or misleading. “Your observability sucks”? Yeah, but that’s life

> “Your observability sucks”? Yeah, but that’s life You could start by asking your AI "help me add better observability to our stack"

Believe it or not, you can’t just run a profiler on everyone’s browser

Re: What Is a Harness?

#189

Post by grey-area »

qarl2 wrote: post #175
grey-area wrote: post #161

Earlier quoted context omitted.

I’ve made my own arcade games with an LLM. It was fun though the code wasn’t great. Mostly I felt they reproduced games from the training data, some with more examples available worked better than others. I use them most days for work, and for that reason don’t trust them that much and certainly don’t worry or fantasise about AGI.

You were making some pretty strong statements about the usefulness of harnesses that seemed to me to be entirely detached from reality. Maybe I misunderstood you? Because I can provide evidence to correct that misconception. Another of my projects is to incorporate Pixar's ideas from RenderMan into a 3d printer slicer. Displacement shaders, in a 3d printer, have never been done before. Would you like to see that? I h…

I’m pleased for you.

In my experience the use of agents and harnesses and other scaffolding around them doesn’t improve the performance of LLMs much, which is adequate for some tasks under supervision but nothing like general intelligence (or electricity for that matter). But good that it works for you.

Re: What Is a Harness?

#190

Post by francisofascii »

Still not sure I get it. Which of the below are harnesses? These all sound like they meet the definiton.

- Codex / VSCode Github Copilot Plugin

- ChatGPT Web application

- Code I wrote to send/process REST API requests to an Antropic / OpenAI API endpoint

Post reply on HN