[flagged]
The agent harness belongs outside the sandbox
51–60 of 140 posts
Re: The agent harness belongs outside the sandbox
#52I am not sure anyone knows what a harness is at this point. I've heard 17 different definitions of it at this point. It's almost like a buzzword in search of a problem.
I don’t even know what an agent means, let alone harness.
I'd say the core is that the harness/runtime/${whatever you call it} doesn't just unconditionally sends model output to the user, and user input to the model, +/- some post-processing, but instead runs a loop that feeds the output back to the model if some conditions are met. That gives you basic "thinking" and single "function calling" a-la early ChatGPT. However, if you allow it to loop arbitrary number of times and allow the output to decide whether to loop or to stop, you get a basic agent.
Re: The agent harness belongs outside the sandbox
#53Slightly related: I am looking for: - Easy single command CLI agent spawning with templates - Automatic context transfer (i. e. a bit like git worktrees) - Fully containerised, but remote (a bit like pods) - Central, mitm-proxy zero trust authn/authz management (no keys or credentials inside the agents), rather enrichment in the hypervisor/encapsulation - Multi agent follow-up functionalities - Fully self hosted/FOSS…
Yeah, have you tried `mngr` by Imbue? It seems to have a bunch of the features you're looking for.
Re: The agent harness belongs outside the sandbox
#54There are other models. Eschew the sandbox. Give the agent a computer, with all the trimmings, but keep that computer segregated from sensitive resources. Tokens are a solved problem: tokenize them[1] or do something equivalent with a proxy. The same thing goes for secrets. A lot of this post presents false dichotomies. It assumes the existence of a sandbox that is by definition ephemeral or "cattle-like". Why? There…
Author here.
This is an interesting and novel field, so I’m not pretending I know the answers, but this is what worked for us :)
At the end of the day, and oversimplifying things: why would I want to spawn a for loop that calls an API (LLM) into its own dedicated sandbox/computer?
When the model wants to run a command, it’ll tell you so. Doesn’t need to be a local exec, you can run it anywhere, the model won’t know the difference.
The agent loop itself doesn’t need sandboxing. In many cases, most tool calls don’t require sandboxing either. For the tools that do require a computer, you can route those requests there when needed, rather than running the whole software in that sandbox.
To me running the agent loop in the sandbox itself feels like “you should run your API in your DB container because it’ll talk to it at some point”.
Re: The agent harness belongs outside the sandbox
#55There are other models. Eschew the sandbox. Give the agent a computer, with all the trimmings, but keep that computer segregated from sensitive resources. Tokens are a solved problem: tokenize them[1] or do something equivalent with a proxy. The same thing goes for secrets. A lot of this post presents false dichotomies. It assumes the existence of a sandbox that is by definition ephemeral or "cattle-like". Why? There…
I'm also very excited by the different shapes for solving problems in this space. A little worried that the path dependence is ACTUALLY a bit warranted since "popular harness engineering is just claude-wrapping" is a bit of a self-fulfilling prophecy today. I've heard many claims that because LLMs are tuned to specific harnesses, we should expect worse performance with novel architectures. That seems to make people r…
Author here.
I’m worried about the same (models tuned for specific harnesses).
We actually work around that by respecting the “contract”. For instance, our harness’ Bash signature is exactly the same as Claude’s. We do our sandboxing stuff and respond using the same format.
In the “eyes” of the model there’s no difference between what Claude does and what we do (even though the implementation is completely different).
We basically use Claude’s tools as API contract
Re: The agent harness belongs outside the sandbox
#56Earlier quoted context omitted.
> Ever since Mitchell Hashimoto mentioned the harness in February What. The idea is as old as anyone can remember, and wrt. LLMs, it was known to be important since at least as early as ChatGPT being first released.
Yes, the concept itself is not new. Around 2022, people would usually have called it the orchestration layer. But I think the term started being used closer to its current meaning around this point: https://www.softwareimprovementgroup.com/blog/what-is-harnes... In a way, the sequence was something like: prompt engineering(23~4) -> context engineering(25) ->harness engineering(26) At first, it was mostly understood a…
Harness itself was a widely used term by at least the "[LLM] plays pokemon" trend, which was a year ago[1]. That was basically the term of art to use when arguing about just how much special treatment LLMs should get.
"harness engineering" is the term claimed by that article to have originated in February. It does seem obvious in retrospect and I don't remember an origination point, but there's at least one hn comment predating that in December[2] and it doesn't treat it as novel.
I will admit that my bias is against any self congratulatory buzzword fads (I'm still not over "MCP is the USB of LLMs" or whatever and that's been a year now too). "Who coined the term harness engineering?" -> who cares? It was already widely being done.
[1] https://www.lesswrong.com/posts/7mqp8uRnnPdbBzJZE/is-gemini-...
Re: The agent harness belongs outside the sandbox
#57I think it omits the real reason I want to run the harness in the sandbox: I barely trust the harness more than the LLM, at least at this point in time. They are so rapidly evolving along with the underlying models, that I don't think they are a reasonable component to rely on to provide safety constraints. Put more precisely: if your harness has an ability to do something the LLM can't, and it has a set of condition…
Author here.
I should have made it more clear that the article is about agent / harness building (not about running third party agents).
> I barely trust the harness more than the LLM
Since we built it, I trust it just as much as I trust our API server :)
The latter gets untrusted inputs from the internet, while the former gets untrusted inputs from the LLM
Re: The agent harness belongs outside the sandbox
#58Earlier quoted context omitted.
Yes, the concept itself is not new. Around 2022, people would usually have called it the orchestration layer. But I think the term started being used closer to its current meaning around this point: https://www.softwareimprovementgroup.com/blog/what-is-harnes... In a way, the sequence was something like: prompt engineering(23~4) -> context engineering(25) ->harness engineering(26) At first, it was mostly understood a…
Harness itself was a widely used term by at least the "[LLM] plays pokemon" trend, which was a year ago[1]. That was basically the term of art to use when arguing about just how much special treatment LLMs should get. "harness engineering" is the term claimed by that article to have originated in February. It does seem obvious in retrospect and I don't remember an origination point, but there's at least one hn commen…
I read your comment. I think we may be talking about slightly different contexts.
The Pokémon article you linked is basically about benchmarking. In that context, the harness functions as part of the benchmark setup: the controlled environment around the model, the available inputs, tools, and assistance.
The current usage of “harness,” at least in the agent engineering discussion, seems closer to a lower-level runtime layer, almost like an OS around the agent.
So I see this as a transition: from “harness” as a narrower benchmark/control-variable layer to “harness” as the broader operating environment of the agent.
That does not mean I think your point is wrong. With topics like this, the interpretation depends on which part of the lineage one emphasizes. The first appearance of the idea may go back to 2022 or earlier, while the usage that looks closer to the current meaning may have emerged at a different point.
I am probably giving more weight to the SIG article, while you are giving more weight to a different point in the lineage. Both seem reasonable to me.
Re: The agent harness belongs outside the sandbox
#59I think it omits the real reason I want to run the harness in the sandbox: I barely trust the harness more than the LLM, at least at this point in time. They are so rapidly evolving along with the underlying models, that I don't think they are a reasonable component to rely on to provide safety constraints. Put more precisely: if your harness has an ability to do something the LLM can't, and it has a set of condition…
I don't trust the harness, and I especially don't trust that the LLM won't be able to subvert the harness, or trick me via the harness. I assume that the LLM will be able to leak any secret in the harness context to arbitrary internet destinations, or somehow encode the secret in a work product. Eg space characters at the end of lines encoding access tokens.
Having the harness in one VM, and tool use applied to user data in another, is about as safe as you can be at present. You can mount filesystem fragments from the data VM into the harness VM, but tool execution remains painful.
Having all authorisation and access control exist outside of the harness layer is essential. It should only have narrowly scoped and time limited credentials that are bound to its IP, and even then that is problematic.
Re: The agent harness belongs outside the sandbox
#60Interesting idea. Tangentially related I’ve been using my local agent to interact with remote shells via zmx, described here: https://bower.sh/zmx-ai-portal
The use case is different but this article strikes some vague similarities around an agent API to remotely execute commands.