Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Superhuman AI Exfiltrates Emails

promptarmor.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Superhuman AI Exfiltrates Emails

#21

Post by djaouen »

TeMPOraL wrote: post #15
djaouen wrote: post #5

Programming used to prevent this by separating code from data. AI (currently) has no such safeguards.

Reality doesn't have a distinction between "code" and "data"; those are categories of convenience, and don't even have a proper definition (what is code and what is data depends on who's asking and why). Any such distinction requires mechanically enforcing it; AI won't have it, because it's not natural, and adding it destroys generality of the model.

OK, then sequence your DNA and send it to me. I will make sure to use it as code!

Re: Superhuman AI Exfiltrates Emails

#23

Post by TeMPOraL »

djaouen wrote: post #21
TeMPOraL wrote: post #15

Earlier quoted context omitted.

Reality doesn't have a distinction between "code" and "data"; those are categories of convenience, and don't even have a proper definition (what is code and what is data depends on who's asking and why). Any such distinction requires mechanically enforcing it; AI won't have it, because it's not natural, and adding it destroys generality of the model.

OK, then sequence your DNA and send it to me. I will make sure to use it as code!

Haha. But DNA is a very good example of what I'm talking about. It's both "code" and "data" at the same time - or rather, a perfect demonstration that these concepts don't exist in nature.

Re: Superhuman AI Exfiltrates Emails

#24

Post by djaouen »

TeMPOraL wrote: post #23
djaouen wrote: post #21

Earlier quoted context omitted.

OK, then sequence your DNA and send it to me. I will make sure to use it as code!

Haha. But DNA is a very good example of what I'm talking about. It's both "code" and "data" at the same time - or rather, a perfect demonstration that these concepts don't exist in nature.

Yes, but for me to use your DNA as code would be a major malfunction!

Re: Superhuman AI Exfiltrates Emails

#25

Post by TeMPOraL »

djaouen wrote: post #24
TeMPOraL wrote: post #23

Earlier quoted context omitted.

Haha. But DNA is a very good example of what I'm talking about. It's both "code" and "data" at the same time - or rather, a perfect demonstration that these concepts don't exist in nature.

Yes, but for me to use your DNA as code would be a major malfunction!

I get the joke, but it's also an incredibly interesting topic to ponder. Remember "Reflections on Trusting Trust"? Now consider that DNA itself needs a complex biomolecular machine to "compile" it into cells and organisms, and that this also embeds in them copies of the "compiler" itself. This raises the question of whether, and how much, information needed to build the organism is not explicitly encoded anywhere in the DNA itself, and instead accumulates in the replication mechanism and gets carried over implicitly.

So for you to successfully use my DNA as code, without also borrowing the compiler from my body, would be a major scientific result, shining light on the questions outlined above.

So in short: I'm happy to contribute my DNA if you cite me as co-author on the resulting paper :P.

Re: Superhuman AI Exfiltrates Emails

#26

Post by majormajor »

TeMPOraL wrote: post #14
SahAssar wrote: post #13

Earlier quoted context omitted.

Employees are under contract and are screened for basic competence. LLMs aren't and can't be.

> Employees are under contract and are screened for basic competence. LLMs aren't So perhaps they should be. > and can't be. Ah but they must, because there's not much else you can do. You can't secure LLMs like they were just regular, narrow-purpose software, because they aren't. They're by nature more like little people on a chip (this is an explicit design goal) - and need to be treated accordingly.

Sooo the primary way we enforce contracts and laws against people are things like fines and jail time.

How would you apply the threat of those to "little people on a chip", exactly?

Imagine if any time you hired someone there was a risk that they'd try to steal everything they could from your company and then disappear forever with you having no way to hold them to account? You'd probably stop hiring people you didn't already deeply trust!

Strict liability for LLM service providers? Well, that's gonna be a non-starter unless there's a lot of MAJOR issues caused by LLMs (look at how little we care about identity theft and financial fraud currently).

Re: Superhuman AI Exfiltrates Emails

#27

Post by stubish »

This demonstrates how adding AI features to software such as web browsers dramatically increases the attack surface. It has to be considered potentially malicious and jailed, and hopefully everyone remembers to respect that jail and put up guardrails. Given our history of chroots and jails and containers and virtualization, we know escapes are going to happen. Reminds me of Word and Excel viruses, when scripting was added to documents and left on by default.

Re: Superhuman AI Exfiltrates Emails

#28

Post by touristtam »

0xferruccio wrote: post #3

The primary exfiltration vector for LLMs is making network requests via images with sensitive data as parameters. As Claude Code increasingly uses browser tools, we may need to move away from .env files to something encrypted, kind of like rails credentials, but without the secret key in the .env

Can't use a tool like dotenvx?

Re: Superhuman AI Exfiltrates Emails

#30

Post by bossyTeacher »

TeMPOraL wrote: post #12
wat10000 wrote: post #9

Earlier quoted context omitted.

Private data, untrusted data, communication: an LLM can safely have two of these, but never all three. Browsing the web is both communication and untrusted data, so it must never have access to any trusted data if it has the ability to browse the web. The problem is, so much of what people want from these things involves having all three.

> The problem is, so much of what people want from these things involves having all three. Pretty much. Also there's no way of "securing" LLMs without destroying the quality that makes them interesting and useful in the first place. I'm putting "securing" in scare quotes because IMO it's fool's errand to even try - LLMs are fundamentally not securable like regular, narrow-purpose software, and should not be treated a…

> I'm putting "securing" in scare quotes because IMO it's fool's errand to even try - LLMs are fundamentally not securable like regular, narrow-purpose software, and should not be treated as such.

Indeed. Between this fundamental unsecurability and alignment, I struggle to see how OpenAI/Anthropic/etc will manage to give their investors enough RoI to justify the investment

Post reply on HN