Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superhuman AI Exfiltrates Emails

promptarmor.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Superhuman AI Exfiltrates Emails

#11

Post by TeMPOraL »

SahAssar wrote: post #4
0xferruccio wrote: post #3

The primary exfiltration vector for LLMs is making network requests via images with sensitive data as parameters. As Claude Code increasingly uses browser tools, we may need to move away from .env files to something encrypted, kind of like rails credentials, but without the secret key in the .env

So you are going to take the untrusted tool that kept leaking your secrets, keep the secrets away from it but still use it to code the thing that uses the secrets? Are you actually reviewing the code it produces? In 99% of cases that's a "no" or a soft "sometimes".

That's exactly what one does with their employees when one deploys "credential vaults", so?

Re: Superhuman AI Exfiltrates Emails

#12

Post by TeMPOraL »

wat10000 wrote: post #9
matusp wrote: post #6

Earlier quoted context omitted.

I wonder how they handled it. Everybody's connecfing their AI to the Web, but it automatically means that any data AI has access to can be extracted by the attacker. The only safe way forward is to 1. disconnect the Web or 2. perhaps to filter the generated URLs aggressively.

Private data, untrusted data, communication: an LLM can safely have two of these, but never all three. Browsing the web is both communication and untrusted data, so it must never have access to any trusted data if it has the ability to browse the web. The problem is, so much of what people want from these things involves having all three.

> The problem is, so much of what people want from these things involves having all three.

Pretty much. Also there's no way of "securing" LLMs without destroying the quality that makes them interesting and useful in the first place.

I'm putting "securing" in scare quotes because IMO it's fool's errand to even try - LLMs are fundamentally not securable like regular, narrow-purpose software, and should not be treated as such.

Re: Superhuman AI Exfiltrates Emails

#13

Post by SahAssar »

TeMPOraL wrote: post #11
SahAssar wrote: post #4

Earlier quoted context omitted.

So you are going to take the untrusted tool that kept leaking your secrets, keep the secrets away from it but still use it to code the thing that uses the secrets? Are you actually reviewing the code it produces? In 99% of cases that's a "no" or a soft "sometimes".

That's exactly what one does with their employees when one deploys "credential vaults", so?

Employees are under contract and are screened for basic competence. LLMs aren't and can't be.

Re: Superhuman AI Exfiltrates Emails

#14

Post by TeMPOraL »

SahAssar wrote: post #13
TeMPOraL wrote: post #11

Earlier quoted context omitted.

That's exactly what one does with their employees when one deploys "credential vaults", so?

Employees are under contract and are screened for basic competence. LLMs aren't and can't be.

> Employees are under contract and are screened for basic competence. LLMs aren't

So perhaps they should be.

> and can't be.

Ah but they must, because there's not much else you can do.

You can't secure LLMs like they were just regular, narrow-purpose software, because they aren't. They're by nature more like little people on a chip (this is an explicit design goal) - and need to be treated accordingly.

Re: Superhuman AI Exfiltrates Emails

#15

Post by TeMPOraL »

djaouen wrote: post #5

Programming used to prevent this by separating code from data. AI (currently) has no such safeguards.

Reality doesn't have a distinction between "code" and "data"; those are categories of convenience, and don't even have a proper definition (what is code and what is data depends on who's asking and why). Any such distinction requires mechanically enforcing it; AI won't have it, because it's not natural, and adding it destroys generality of the model.

Re: Superhuman AI Exfiltrates Emails

#16

Post by SahAssar »

TeMPOraL wrote: post #14
SahAssar wrote: post #13

Earlier quoted context omitted.

Employees are under contract and are screened for basic competence. LLMs aren't and can't be.

> Employees are under contract and are screened for basic competence. LLMs aren't So perhaps they should be. > and can't be. Ah but they must, because there's not much else you can do. You can't secure LLMs like they were just regular, narrow-purpose software, because they aren't. They're by nature more like little people on a chip (this is an explicit design goal) - and need to be treated accordingly.

> So perhaps they should be.

Unless both the legalities and technology radically change they will not be. And the companies building them will not take on the burden since the technology has proved to be so unpredictable (partially by design) and unsafe.

> designed to be more like little people on a chip - and need to be treated accordingly

Deeply unpredictable and unsafe people on a chip, so not the sort that I generally want to trust secrets with.

I don't think it's that complex, you can have secure systems or you can have current gen LLMs. You can't have both in the same place.

Re: Superhuman AI Exfiltrates Emails

#17

Post by ineedasername »

Why does an agent tasked with email summarizing have access to anything else? There’s plenty of difference between an agent and a background service or daemon but it’s at minimum got to be given the same restrictions in scope they would be, or an intern using your system for the same purpose. Developers need to bring the same ZTA mindset to agent permissions they would to building the other services and infrastructure they rely on.

Re: Superhuman AI Exfiltrates Emails

#18

Post by TeMPOraL »

SahAssar wrote: post #16
TeMPOraL wrote: post #14

Earlier quoted context omitted.

> Employees are under contract and are screened for basic competence. LLMs aren't So perhaps they should be. > and can't be. Ah but they must, because there's not much else you can do. You can't secure LLMs like they were just regular, narrow-purpose software, because they aren't. They're by nature more like little people on a chip (this is an explicit design goal) - and need to be treated accordingly.

> So perhaps they should be. Unless both the legalities and technology radically change they will not be. And the companies building them will not take on the burden since the technology has proved to be so unpredictable (partially by design) and unsafe. > designed to be more like little people on a chip - and need to be treated accordingly Deeply unpredictable and unsafe people on a chip, so not the sort that I gene…

> Deeply unpredictable and unsafe people on a chip, so not the sort that I generally want to trust secrets with.

Very true when comparing to acquaintances, but at a scale of any company or system except the tiniest ones, you can't blindly trust people in general either. Building systems involving people and LLMs is pretty similar.

> I don't think it's that complex, you can have secure systems or you can have current gen LLMs. You can't have both in the same place.

That is, indeed, the key. My point is that, unlike the popular opinion in threads like this, it does not follow that we need to give up on LLMs, or that we need to fix the security issues. The former is undesirable, the latter is fundamentally impossible.

What we need is what we've been doing ever since civilization took shape, ever since we've started building machines: recognize that automatons and people are different kinds of components, with different reliability and security characteristics. You can't blindly substitute one for the other, but there are ways to make them work together. Most systems we've created are of that nature.

What people still get wrong is treating LLMs as "automatons" components. They're not, they're "people" components.

Re: Superhuman AI Exfiltrates Emails

#19

Post by SahAssar »

TeMPOraL wrote: post #18
SahAssar wrote: post #16

Earlier quoted context omitted.

> So perhaps they should be. Unless both the legalities and technology radically change they will not be. And the companies building them will not take on the burden since the technology has proved to be so unpredictable (partially by design) and unsafe. > designed to be more like little people on a chip - and need to be treated accordingly Deeply unpredictable and unsafe people on a chip, so not the sort that I gene…

> Deeply unpredictable and unsafe people on a chip, so not the sort that I generally want to trust secrets with. Very true when comparing to acquaintances, but at a scale of any company or system except the tiniest ones, you can't blindly trust people in general either. Building systems involving people and LLMs is pretty similar. > I don't think it's that complex, you can have secure systems or you can have current…

I think I generally agree, but I also think that treating them like people means that you expect reason, intelligence and a way to interrogate their way of "thinking" (very broad quotes here).

I think LLMs are to be treated as something completely separate from both predictable machines ("automatons") and people. They have separate concerns and fitness for a use-case than both existing categories.

Re: Superhuman AI Exfiltrates Emails

#20

Post by rapind »

ineedasername wrote: post #17

Why does an agent tasked with email summarizing have access to anything else? There’s plenty of difference between an agent and a background service or daemon but it’s at minimum got to be given the same restrictions in scope they would be, or an intern using your system for the same purpose. Developers need to bring the same ZTA mindset to agent permissions they would to building the other services and infrastructur…

“Move fast and break things.” It’s funny you even need to ask on hacker news of all places. ;)
Post reply on HN