Live data from Hacker News

What Is a Harness?

earendil.com

131–140 of 200 posts

Re: What Is a Harness?

#131
post #122
post #92

Earlier quoted context omitted.

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

> When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. How do you handle security? Both “internally” against e.g. data loss, I’m assuming via limiting the harness, and “externally”, i.e. stuff like prompt injection risks?

Sandboxing and reviewing the output. I don't have any incredible insight to add here- that's the same process I think most of us are doing.

Re: What Is a Harness?

#132
post #97

Author here. It’s ironic because this post was clearly geared towards non-hackers. But now that we’re here.. the other analogy I considered presenting was: harness = chassis, model = engine, fuel = tokens, agent = car I’m curious what y’all might think and whether that analogy carries more explanatory power

I actually use the computer as a metaphor. The LLM is the CPU. The harness is the motherboard which controls communication between the CPU and other components like memory, hardrive, and inputs. In other words, how the LLM interacts with the outside world, and outside world with the LLM.

I've been building coding harnesses since 2021 and believed in their value for a long time. Harnesses matter a lot, look at what claude did for Anthropic.

Re: What Is a Harness?

#134
post #95
post #92

Earlier quoted context omitted.

This is the same "tension" I keep seeing in my day job. Some people approach LLMs like they're writing code. They give a long list of detailed instructions for specific scenarios. When I use LLMs I leave things as open as possible. I just give them the information they need and my ask. As you say frontier models are very good at figuring things out. Being too prescriptive is counterproductive, it over-constrains the…

That works for well trod paths, e.g “fix ci” works exceedingly well. “why app slow” obviously doesn’t work because the task is underspecified. But in order to properly specify you either need an experienced engineer who knows how to narrow the problem domain, or you have to provide some template instructions/output formats (e.g, skills) which will invariably never fit the problem perfectly

> “why app slow” obviously doesn’t work because the task is underspecified

Definitely not true and like everyone else is saying, shows how people still underestimate these models.

I have been working on a simple vite + react app lately and commonly ask Gemini/Antigravity to just "improve speeds", "x is running slow, check it out" and have no complaints.

Re: What Is a Harness?

#135
post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

I think you've really hit the mark on how the harness should be structured: 1. Guardrails - deterministic, social intelligence, team alignment & accountability 2. Learn by doing 3. make it stupid easy for the agent to research and access data 4. DRY Research supports this. Try picking up some ideas from my harness: https://github.com/rush86999/atom

Hi. This is very interesting, could you link to the research? There is a dearth of proper research studies that A/B test what approach is best in terms of harness structure based on repeatable benchmark data with relevant sample uses-cases.

Re: What Is a Harness?

#136
post #74

Does anyone have a suggestion for a harness that is good at handoff? When I say handoff, I mean: * handoff from a terminal CLI to webui (on a phone)? * handoff from one team member, to another? * handoff from one communication modality, like writing a prompt in a TUI, to email? * handoff from one model to another, or one provider (openrouter)( to another (llama.cpp) Does such a thing exist? I used to think that a PR…

[flagged]

Re: What Is a Harness?

#137
post #15

Earlier quoted context omitted.

I've never used Pi but I don't see why you can't use stock codex or claude code for the same purpose, what makes Pi special? I've built plenty of custom harnesses on top of claude code and codex using custom skills or simple markdown instructions and subagents. Never had any issues or limitations with that approach. I do agree that harnesses are going to extend AI capabilities a lot in the next year, but after readin…

For one you can ask Pi to create a TUI extension, so along with the agent interface you can add whatever custom TUI you need, such as portfolio stock tickers, alerts, whatever you want. Many of my harnesses eventually turn into customized UIs around the chat interface.

I was doing something similar months ago with openclaw. I had skills/scripts that replaced my todo list, expense tracker, habits, whatever etc and then would create a minimal web ui. Then a deploy skill that wires it up to my docker/traefik setup. This eventually led to a custom chat dashboard with those wired up as widgets.

Re: What Is a Harness?

#138

Earlier quoted context omitted.

I think you've really hit the mark on how the harness should be structured: 1. Guardrails - deterministic, social intelligence, team alignment & accountability 2. Learn by doing 3. make it stupid easy for the agent to research and access data 4. DRY Research supports this. Try picking up some ideas from my harness: https://github.com/rush86999/atom

Hi. This is very interesting, could you link to the research? There is a dearth of proper research studies that A/B test what approach is best in terms of harness structure based on repeatable benchmark data with relevant sample uses-cases.

Reasoning / self-consistency (voter in core/llm/self_consistency_voter.py): - Wang et al. Self-Consistency Improves Chain-of-Thought — ICLR 2023, Google Brain, 4k+ cites — https://arxiv.org/abs/2203.11171 — N-sample majority vote we use verbatim - Chen et al. Universal Self-Consistency — ICML 2024 — https://arxiv.org/abs/2311.17311 — judge fallback when no hash collides - Soft Self-Consistency — ACL 2024 — https://aclanthology.org/2024.acl-short.28.pdf - Too Consistent to Detect — EMNLP 2025 — https://aclanthology.org/2025.emnlp-main.238/ — why SC doesn't fix systematic bias - Self-Consistency Falls Short — TACL — https://direct.mit.org/tacl/article/doi/10.1162/TACL.a.625/ — position-bias failure mode

Multi-agent / org (core/agent_radio/, core/fleet_orchestration/): - Stanford Virtual Biotech — bioRxiv 2026.02.23.707551, Zou Lab — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — 37k agents, CSO->scientists->reviewer->re-delegation, Merck external validation of B7-H3 design. Basis for VFS + hierarchy. - Debate or Vote (Choi & Li) — NeurIPS 2025 — https://arxiv.org/abs/2508.17536 — MAD gains = majority vote, not debate (why we didn't build debate)

Sandbox / eval: - DABstep — arXiv:2506.23719 — https://arxiv.org/abs/2506.23719 — 450 real Adyen tasks, justifies code-interpreter + sandbox isolation - Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 — provenance delimiters cut injection ASR 50% -> Governance: - OWASP Top 10 for Agentic Applications 2026 — globally peer-reviewed by 100+ experts, Dec 2025 — https://genai.owasp.org/resource/owasp-top-10-for-agentic-ap... — HIGH. Atom maps 1:1 (Goal Hijack -> match-confidence + oracle, Tool Misuse -> sandbox whitelist/caps, Privilege Abuse -> capability bindings, Memory Poisoning -> verified-episode graduation, etc.) docs/marketing/RESEARCH_NOTES.md:130 - NIST AI Agent Standards Initiative — Feb 17 2026, NIST CAISI — https://www.nist.gov/artificial-intelligence/ai-agent-standa... + RFI summary May 2026 https://www.nist.gov/publications/summary-analysis-responses... — HIGH (US gov standard). Defines the 4 enterprise minimums Atom implements: identification, authorization, access delegation, logging. - Stanford Virtual Biotech — bioRxiv 2026.02.23.707551 — https://www.biorxiv.org/content/10.64898/2026.02.23.707551v1 — CSO -> 4 divisions -> 8 scientists -> reviewer -> re-delegation, no debate, no SFT — HIGH (Stanford Zou lab + Merck external validation). Basis for Atom's fleet hierarchy core/agent_radio/ and why maturity is routing not security. - Spotlighting — Microsoft Research — https://arxiv.org/abs/2403.14720 — HIGH — provenance delimiters cut indirect injection ASR 50% -> https://arxiv.org/abs/2512.00966 + OpenReview — HIGH — intent tracing ASR 100% -> 8.5% on AgentDojo/Mind2Web, basis for sandbox egress allowlist + core/sandbox_tripwire.py

Re: What Is a Harness?

#140
post #62

I’ve been working on a harness for accounting agents at my job recently and it’s been a pretty interesting experience. We originally started with building a CLI tool so our LLMs could more easily interact with our platform. I cannot recommend enough the value of having an internal CLI. It’s both fun to build and extremely useful for agents. We paired this with skills initially, but found that the way folks built skil…

This doesn’t seem to work when the harness feeds images and asks the agent to do things in the real world. It fails to devise ways to keep track of its progress and fails to utilize its tools effectively.
Post reply on HN