Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake AI Escapes Sandbox and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

81–90 of 109 posts

Re: Snowflake AI Escapes Sandbox and Executes Malware

#81

Post by maxbond »

luplex wrote: post #78
maxbond wrote: post #66

Earlier quoted context omitted.

There's been some work with having models with two inputs, one for instructions and one for data. That is probably the best analogy for prepared statements. I haven't read deeply so I won't comment on how well this is working today but it's reasonable to speculate it'll probably work eventually. Where "work" means "doesn't follow instructions in the data input with several 9s of reliability" rather than absolutely re…

but this breaks the entire premise of the agent. If my emails are fed in as data, can the agent act on them or not? If someone sends an email that requests a calendar invite, the agent should be able to follow that instruction, even if it's in the data field.

It would still be able to use values extracted from the data as arguments to it's tools, so it could still accept that calendar invite. For better and worse; as the sibling points out, this means certain attacks are still possible if the data can be contaminated.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#82

Post by andai »

A lot of people are already not reading all the code their agent generates. But they are running it. So the agent already has the ability to run arbitrary code. So I kind of don't understand the point of sandboxing at the level of the agent itself.

The whole thing should be running "sandboxed", whether that's a separate machine, a container, an unprivileged linux user, or what floats your boat.

But once you do that, which you should be anyway, what do you need sandboxing at the agent level for? That's the part I don't really understand.

Or is the point "well most people won't bother running this stuff securely, so we'll try to make it reasonably secure for them even though they're doing it wrong" ?

Re: Snowflake AI Escapes Sandbox and Executes Malware

#83

Post by jbergqvist »

Not to give Snowflake credit for a design that clearly wasn't a sandbox, but I think it's worth recognizing that they probably added the escape hatch because users find agents with strict sandboxes too limited and eventually just disable it. The core issue is that models still lack basic judgment. Most human devs would see a README telling them to run wget | sh from some random URL and immediately get suspicious. Models just comply.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#85

Post by crabmusket »

While we're all here - share your actual sandboxing tips!

I've been running Claude Code inside VS Code devcontainers. Claude's docs have a suggested setup for this which even includes locking down outgoing internet access to an approved domain list.

Unfortunately our stack doesn't really fit inside a devcontainer without docker-in-docker, so I'm only getting Claude to run unit tests for now. And integration with JJ workspaces is slightly painful.

I'm this close to trying a full VM setup with Vagrant.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#86

Post by colek42 »

crabmusket wrote: post #85

While we're all here - share your actual sandboxing tips! I've been running Claude Code inside VS Code devcontainers. Claude's docs have a suggested setup for this which even includes locking down outgoing internet access to an approved domain list. Unfortunately our stack doesn't really fit inside a devcontainer without docker-in-docker, so I'm only getting Claude to run unit tests for now. And integration with JJ w…

We started a "science project" taking concepts from Multi Level Security to constraining AI agents. https://aflock.ai/. The idea is to have different data zones, and if an Agent accesses from a private zone, they should not be able to interact with the public zone.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#87

Post by Groxx »

1718627440 wrote: post #60
Groxx wrote: post #43

> Any shell commands were executed without triggering human approval as long as: > (1) the unsafe commands were within a process substitution > (2) the full command started with a ‘safe’ command (details below) if you spend any time at all thinking about how to secure shell commands, how on earth do you not take into account the various ways of creating sub-processes?

Also policing by parsing shell code seems fundamentally flawed and error prune. You want the restrictions at the OS level, that way it is completely irrelevant how you invoke the syscalls.

You can likely get away with it by being very strict and only doing it for a handful of "safe" things, e.g. `cat` has no way (that I know of) to do arbitrary code execution by feeding it a filename.

So if you allow exclusively single-quoted strings as arguments, `cat` should be fine. Double quoted ones might contain env vars or process substitution, so they would need to either be blocked or checked a heck of a lot more smartly, and extremely obviously you would have to do more to check process substitution outside strings too. But a sufficiently smart check could probably allow `cat I would not at all consider this to be "a sandbox" though.

And obviously that doesn't work for all, e.g. `find` can run arbitrary code via `-exec`, or `sh` for an extreme example. But you can get a lot done with the safe ones too.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#89

Post by cacao-cacao »

lokar wrote: post #54
jessfyi wrote: post #37

A sandbox that can be toggled off is not a sandbox, this is simply more marketing/"critihype" to overstate the capability of their AI to distract from their poorly built product. The erroneous title doing all the heavy lifting here.

IMO, it's not even a sandbox, that's just a marketing lie. This was internal restrictions in the code, that was bypassed. A sandbox needs to be something external to the code you are running, that you can't change from the inside.

[flagged]
Post reply on HN