Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The agent harness belongs outside the sandbox

mendral.com

Post reply on HN
Print view View original

91–100 of 140 posts

Re: The agent harness belongs outside the sandbox

#91

Post by bauerd »

zmmmmm wrote: post #30

I think it omits the real reason I want to run the harness in the sandbox: I barely trust the harness more than the LLM, at least at this point in time. They are so rapidly evolving along with the underlying models, that I don't think they are a reasonable component to rely on to provide safety constraints. Put more precisely: if your harness has an ability to do something the LLM can't, and it has a set of condition…

>Having said that, some components need to live outside the sandbox (otherwise, who creates the sandbox?).

I run a single-node k3d cluster on each of my MacBooks which uses Agent Sandbox[0] to keep harnesses isolated. Harnesses access models through LiteLLM only. I have aliases for `kubectl exec`ing into whatever harness I need.

[0] https://agent-sandbox.sigs.k8s.io

Re: The agent harness belongs outside the sandbox

#92

Post by ramraj07 »

tptacek wrote: post #42

There are other models. Eschew the sandbox. Give the agent a computer, with all the trimmings, but keep that computer segregated from sensitive resources. Tokens are a solved problem: tokenize them[1] or do something equivalent with a proxy. The same thing goes for secrets. A lot of this post presents false dichotomies. It assumes the existence of a sandbox that is by definition ephemeral or "cattle-like". Why? There…

> It assumes the existence of a sandbox that is by definition ephemeral or "cattle-like". Why?

Because the moment you use k8s, you have to assume that, apparently. Or so Im told by all the infrastructure people I speak with. Getting these pods to not disappear just because one process ran out of memory has been an herculean task.

I wish our standard deploy processes produce durable computers that dont break our bank but that hasn't been an easy requirement with simple infra teams.

Re: The agent harness belongs outside the sandbox

#93

Post by ramraj07 »

skybrian wrote: post #24

They didn't make a clear argument in favor of that architecture and I'm not really convinced. On exe.dev the agent (Shelley) runs in a Linux VM, which is the security boundary. All the conversations are saved to a sqlite database, and it knows how to read it, so you can refer to a previous conversation in the database. It's also handy for asking the AI to do random sysadmin stuff, since it can use sudo. A downside is…

No matter how smart you think you get, I personally dont trust the models in an environment where they can read the secrets one way or another, in any high volume production environment.

Re: The agent harness belongs outside the sandbox

#94

Post by ramraj07 »

spankalee wrote: post #29

This is angling in the right direction, but I think it has two problems: 1) It's still assuming agents have CLIs. This is a very developer-centric concept of agents, and doesn't map well to either consumer or enterprise agents that aren't primarily working with files. Skills, plans, TODO lists, and memory are good, but don't have to be modeled as raw file access. Many harnesses have tools for them. 2) It's talking ab…

For 1, the general thinking is that companies like these perform the job of abstracting the CLI complexity in their application while the harness presented to the llm can be independently as suave as needed for it.

Re: The agent harness belongs outside the sandbox

#98

Post by schaefer »

To me the article isn’t convincing. There are some interesting points raised. Like “the llm sharing memories between a team of developers”.

I mostly use llms individually, so this is a real blind spot for me. But I might convinced to share a corpus of memories between developers if it ever becomes practical.

But for now aren’t context windows still sometimes smaller than the task at hand… so those “llm memories” take the form of literal documentation???

The closer llm memories get to native format, i.e: stored tokenized content, don’t we lose compatibility between setups anyhow. What if you’re using fp16, and I’m using nvfp4?

Re: The agent harness belongs outside the sandbox

#100

Post by lmeyerov »

We did this from the earliest days for louie.ai, which is an adjacent space of Investigations. Sandboxing the LLM was secondary to the primary reason: the threat model for servers. I suspect most people building agentic products are in this bucket.

Sufficiently advanced desktop tools starts to want server capabilities like teleport, scheduled tasks, ci mode, shared sessions, etc. Web-based ones start here to begin with.

Pretty soon after you have a server, you also think about multitenancy isolation and task isolation. The article's sandboxing also matters for regular old non-LLM code escapes in a multitenant world. We have to assume malicious python by the attacker, whether AI or human, and cannot let one tenant's python have write access to trusted surface of another.

Post reply on HN