Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is a Harness?

earendil.com

Post reply on HN
Print view View original

191–200 of 200 posts

Re: What Is a Harness?

#191

Post by qarl2 »

grey-area wrote: post #189
qarl2 wrote: post #175

Earlier quoted context omitted.

You were making some pretty strong statements about the usefulness of harnesses that seemed to me to be entirely detached from reality. Maybe I misunderstood you? Because I can provide evidence to correct that misconception. Another of my projects is to incorporate Pixar's ideas from RenderMan into a 3d printer slicer. Displacement shaders, in a 3d printer, have never been done before. Would you like to see that? I h…

I’m pleased for you. In my experience the use of agents and harnesses and other scaffolding around them doesn’t improve the performance of LLMs much, which is adequate for some tasks under supervision but nothing like general intelligence (or electricity for that matter). But good that it works for you.

> In my experience the use of agents and harnesses and other scaffolding around them doesn’t improve the performance of LLMs much

I have to say - that seems straight-up crazy to me. Would you mind digging into that discussion?

As just one example - in a harness I can ask an agent to confirm everything it says via a second sub agent - which dramatically improves the output. It almost entirely solves the problem of hallucinations. You don't see that as an improvement?

Re: What Is a Harness?

#192

Post by okamiueru »

sph wrote: post #54
jascha_eng wrote: post #4

The ai hype word for 2026 after agent in 2025 for any LLM powered application. Well kind of, I wouldn't be surprised to see that some things marketed as agents are actually good old deterministic software.

It's really funny (and a bit obnoxious) to watch the vocabulary from the outside. In 2023 everybody learned the word moat , then it's been agent(ic) , from last year there's more talking about harnesses than at a bungee jumping convention. The mot du jour is frontier . It truly proves like there's a handful of thought leaders on Twitter that everybody follows blindly and start to copy down to the lexicon and parrot e…

I find the vocabulary used to be disturbingly fascinating, to the degree concepts are being anthropomorphized. It's so pervasive that I cannot help but assume it is entirely deliberate.

The principles by which LLMs functions haven't changed in the last four years. It is still a next-word-predictor, a statistical parrot, if you will. But if you don't understand the mechanisms behind it, you cannot be faulted for thinking this is something much more. Most of it, is pretty devious marketing.

As an example, no LLM model does anything that can be considered "reasoning", or "intelligent" in the traditional sense, but these words are used extensively. "High reasoning model" is a pricing tier. The article in question has an anthropomorphized term in every single sentence. I'll pick a paragraph at random and highlight the cases. If both parties understand the mechanisms, these words are fine, and we do that all that time. The issue is when one sides is mislead to believe that these systems can be relied on in a way that they should not be, leading to people getting hurt.

> > The *translation layer* is what *allows* a *harness* to *work* with different AI models. In some cases, a *harness may decide* to *use* different models within the same *agentic loop*, because different AI *models may excel* at different tasks. The *translation layer* is also a crucial aspect of *harnesses* because they *deliver control* to the end user. It means that someone can take their *AI harness* and use it with a model from Anthropic, or OpenAI, or explore one of the open weight AI models that often deliver great value-for-money (measured by cost-per-task).

The underlying logic isn't remotely as mysterious or mystic as the language makes it seem. A different paragraph:

> > Tools are a set of *capabilities*, written in code, that the model can *“call”*. The *harness describes* the tools and also *provides* the software that is the tool itself. Examples of these tools might include a web search tool, a tool that *allows the model* to write and execute software code, or a tool that *allows the model* to *compose* an email. Critically, the *harness usually* does not *dictate* when and how the *AI* model should *use* the tool. Instead, it simply *makes* the tools available, *describes* them clearly, and *allows* the *AI* model itself to *decide* when and how *it should use* them

Re: What Is a Harness?

#193

Post by ni10c »

troyvit wrote: post #106
ni10c wrote: post #97

Author here. It’s ironic because this post was clearly geared towards non-hackers. But now that we’re here.. the other analogy I considered presenting was: harness = chassis, model = engine, fuel = tokens, agent = car I’m curious what y’all might think and whether that analogy carries more explanatory power

I'm a climber so I'm biased but I really liked your climbing harness example because of the configuration you're able to easily make to the harness. Saying the harness is like a car's chassis doesn't work as well for me because the chassis isn't as configurable as a climbing harness for as little work. Getting deeper into the climbing analogy you can even swap out the harnesses themselves for wildly different climbs.…

Thanks for these thoughts. Super insightful and appreciated

Re: What Is a Harness?

#194

Post by arw0n »

xrd wrote: post #74

Does anyone have a suggestion for a harness that is good at handoff? When I say handoff, I mean: * handoff from a terminal CLI to webui (on a phone)? * handoff from one team member, to another? * handoff from one communication modality, like writing a prompt in a TUI, to email? * handoff from one model to another, or one provider (openrouter)( to another (llama.cpp) Does such a thing exist? I used to think that a PR…

I found both Tmux and Herdr to be insufferable, but they work for many people. I'm not in the business of managing 17 agents at the same time (I have 1 to 4 agents I manage, those might have sub-agents, but I'm not a micro-manager). The hand-off problem itself is trivial, it is a txt file that needs to be sent around, but I haven't seen any good tools giving good UX for it yet.

1. Means running somewhere that is not your computer. I just use my personal server for this.

2. Should probably be done in the repo. Just add the context file in a dedicated sub directory. At this point I think these kinds of artefacts are part of the SE process, and fair to check into source control.

3. Can be done by letting the TUI write/draft emails via MCP.

4. Can be done with something like OpenCode or PiCode, which just allows you to switch in their TUI interface. Beware though, context has to be re-ingested, so it costs a couple of cents.

But again, in the end context is just a file. By switching models, you are paying the price of not having any context cached, but for manual steps that's usually not super relevant. Just be careful about adding these kinds of things to automated workflows, as a cost like 25c can add up when agents run amok.

Re: What Is a Harness?

#195

Post by sysfiend »

Model -> brain cells' conections Harness -> everything else

I've been working with different setups in parallel for months (openclaw, pi, cursor per project harnesses and codex) and, even though using the same model most of the time, I can clearly see how the behave in very different ways deppending on the setup.

As a language model, language is our way to communicate and build everything around the models, which makes my younger self (who loved writing stories) very very happy :)

Re: What Is a Harness?

#196

Post by stymaar »

theptip wrote: post #183
stymaar wrote: post #164

Earlier quoted context omitted.

The problem is that even Fable still make trivial yet high impact mistake when let on their own, and then you'd need to read the whole code to catch them… Meanwhile they are very good at implementating an explicit algorithm that you feed it to them.

The trick is to set up the harness so that the solution is easy to verify - you’ve profited as long as verification is cheaper than building, but ideally verification is close to automatic (not always achievable of course). Generally you want to include objective/repeatable outputs as citations. An example would be, if you invest in an awesome layered test rig (browser test, fuzz/property tests, very well reviewed un…

> then you should be able to add features by just reading the acceptance test and scanning unit tests.

That “just” is bearing a lot of weight though as tests are often even longer than the code itself, in addition to being excruciating to review.

Re: What Is a Harness?

#197

Post by mijowi »

johnmw wrote: post #143
asQuirreL wrote: post #105

Earlier quoted context omitted.

The first analogy that comes to mind, growing out of "harness", is more like harness = harness, model = horse (rather than harness as in climbing harness). I guess you could say that tokens = hay, and agent = horse and cart, from there? Not sure how useful the hay part is but you could observe from the second that there are many different things you could harness a horse to (also a plough, or a coach, or just a saddl…

I wonder if I'm the only person whose first analogy that came to mind was: model = toddler, harness = baby bouncing jumper harness

Although obviously unserious, I would argue a baby harness is mostly the opposite of what an agent harness does. A baby harness constrains and contains the baby, while an agent harness controls but also empowers the agent.

Re: What Is a Harness?

#198

Post by qarl2 »

grey-area wrote: post #189
qarl2 wrote: post #175

Earlier quoted context omitted.

You were making some pretty strong statements about the usefulness of harnesses that seemed to me to be entirely detached from reality. Maybe I misunderstood you? Because I can provide evidence to correct that misconception. Another of my projects is to incorporate Pixar's ideas from RenderMan into a 3d printer slicer. Displacement shaders, in a 3d printer, have never been done before. Would you like to see that? I h…

I’m pleased for you. In my experience the use of agents and harnesses and other scaffolding around them doesn’t improve the performance of LLMs much, which is adequate for some tasks under supervision but nothing like general intelligence (or electricity for that matter). But good that it works for you.

OH - I think I understand now. You're like LeCun - dismissive of LLMs in all their forms. So of course agents can't lead to AGI, because LLMs just can't.

Which I've always found so strange. Language was always considered the pinnacle of the human mind - right up until we created LLMs. Now it's the world model - the things that animals always had and the thing we used to look down on.

From my perspective, language is still at the top. I'm not a fickle lover of the gaps.

Re: What Is a Harness?

#199

Post by pelasaco »

YZF wrote: post #92
Syntaf wrote: post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

> Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask.

Hm, but thats ok right? I mean some people like to code with LLM and other people like to let LLM code for them.. no?

Re: What Is a Harness?

#200

Post by xrd »

gf000 wrote: post #181
xrd wrote: post #74

Does anyone have a suggestion for a harness that is good at handoff? When I say handoff, I mean: * handoff from a terminal CLI to webui (on a phone)? * handoff from one team member, to another? * handoff from one communication modality, like writing a prompt in a TUI, to email? * handoff from one model to another, or one provider (openrouter)( to another (llama.cpp) Does such a thing exist? I used to think that a PR…

Not really solving all your cases, but I found tmux (or herdr) on a home server works pretty well. I just ssh into my home server and continue where I left off with the same claude/opencode open. On a longer term, I think "assistant" style harnesses might help here, like vellum.ai. I no longer use that, but I asked it to create an ACP proxy through iroh (basically tailscale but on the application layer), and it manag…

I basically do this, tmux via tailscale. But, I often get inspiration and want to jump into a webapp from my phone, or review.

I've been playing with pi and the remote webui extension. I don't love it; it has a lot of chrome that obscures what I want to do. I just want a simple way to review the progress so far, and keep tweaking with minimal setup. Then, jump back into tmux when I'm back on my computer.

Thanks for your comments.

Post reply on HN