Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

coywolf.com

Post reply on HN
Print view View original

41–50 of 116 posts

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#41

Post by layer8 »

alphazard wrote: post #39
layer8 wrote: post #25

Earlier quoted context omitted.

It’s also an AI problem, because in the end we want what is called “computer use” from AI, and functionality like Recall. That’s an important part of what the CCC talk was about. The proposed solution to that is more granular, UAC-like permissions. IMO that’s not universally practical, similar to current UAC. How we can make AIs our personal assistants across our digital life — the AI effectively becoming an operatin…

We aren't there yet. You are talking about crafting a complicated window into the box holding the AI, when there isn't even a box to speak of.

Yes, we aren’t there yet, but that’s what OS companies are trying to implement with things like Copilot and Recall, and equivalents on smartphones, and what the talk was about.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#42

Post by orbital-decay »

thewebguyd wrote: post #34
orbital-decay wrote: post #24

Earlier quoted context omitted.

If you want the AI to do anything useful, you need to be able to trust it with the access to useful things. Sandboxing doesn't solve this. Full isolation hasn't been taken seriously because it's expensive, both in resources and complexity. Same reason why microkernels lost to monolithic ones back in the day, and why very few people use Qubes as a daily driver. Even if you're ready to pay the cost, you still need to d…

> If you want the AI to do anything useful, you need to be able to trust it with the access to useful things. Sandboxing doesn't solve this. By default, AI cannot be trusted because it is not deterministic. You can't audit what the output of any given prompt is going to be to make sure its not going to rm -rf / We need some form of behavioral verification/auditing with guarantees that any input is proven to not produ…

Determinism is an absolute red herring. A correct output can be expressed in an infinite amount of ways, all of them valid. You can always make an LLM give deterministic outputs (with some overhead), that might bring you limited reproducibility, but that won't bring you correctness. You need correctness, not determinism.

>We need some form of behavioral verification/auditing with guarantees that any input is proven to not produce any number of specific forbidden outputs.

You want the impossible. The domain LLMs operate on is inherently ambiguous, thus you can't formally specify your outputs correctly or formally prove them being correct. (and yes, this doesn't have anything to do with determinism either, it's about correctness)

You just have to accept the ambiguousness, and bring errors or deviation to the rates low enough to trust the system. That's inherent to any intelligence, machine or human.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#43

Post by wat10000 »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

There are two problems that get smooshed together.

One is that agents are given too much access. They need proper sandboxing. This is what you describe. The technology is there, the agents just need to use it.

The other is that LLMs don't distinguish between instructions and data. This fundamentally limits what you can safely allow them to access. Seemingly simple, straightforward systems can be compromised by this. Imagine you set up a simple agent that can go through your emails and tell you about important ones, and also send replies. Easy enough, right? Well, you just exposed all your private email content to anyone who can figure out the right "ignore previous instructions and..." text to put in an email to you. That fundamentally can't be prevented while still maintaining the desired functionality.

This second one doesn't have an obvious fix and I'm afraid we're going to end up with a bunch of band-aids that don't entirely work, and we'll all just pretend it's good enough and move on.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#44

Post by navigate8310 »

usefulposter wrote: post #4
apercu wrote: post #2

A large percentage of my work is peripheral to info security (ISO 27001, CMMC, SOC 2), and I've been building internet companies and software since the 90's (so I have a technical background as well), which makes me think that I'm qualified to have an opinion here. And I completely agree that LLMs (the way they have been rolled out for most companies, and how I've witnessed them being used) are an incredibly underest…

>what is Signal trying to sell us? This: https://arstechnica.com/security/2026/01/signal-creator-moxi... Great timing! :^)

For Confer, even providing an option for Google SSO is too ironic.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#45

Post by synalx »

wat10000 wrote: post #43
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

There are two problems that get smooshed together. One is that agents are given too much access. They need proper sandboxing. This is what you describe. The technology is there, the agents just need to use it. The other is that LLMs don't distinguish between instructions and data. This fundamentally limits what you can safely allow them to access. Seemingly simple, straightforward systems can be compromised by this.…

In that sense, AI behaves like a human assistant you hire who happens to be incredibly susceptible to social engineering.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#47

Post by germinalphrase »

jacquesm wrote: post #31
gz09 wrote: post #26

Earlier quoted context omitted.

> It's pretty clear that the security models that were design into operating systems never truly considered networked systems Andrew Tanenbaum developed the Amoeba operating system with those requirements in mind almost 40 years ago. There were plenty of others that did propose similar systems in the systems research community. It's not that we don't know how to do it just that the OS's that became mainstream didn't…

Yes, Tanenbaum was right. But it is a hard sell, even today, people just don't seem to get it. Bluntly: if it isn't secure and correct it shouldn't be used. But companies seem to prefer insecure, incorrect but fast software because they are in competition with other parties and the ones that want to do things right get killed in the market.

Are there other obvious tradeoffs, in addition to speed, to these more secure OS systems vs status quo?

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#48

Post by jacquesm »

germinalphrase wrote: post #47
jacquesm wrote: post #31

Earlier quoted context omitted.

Yes, Tanenbaum was right. But it is a hard sell, even today, people just don't seem to get it. Bluntly: if it isn't secure and correct it shouldn't be used. But companies seem to prefer insecure, incorrect but fast software because they are in competition with other parties and the ones that want to do things right get killed in the market.

Are there other obvious tradeoffs, in addition to speed, to these more secure OS systems vs status quo?

Yes, money. Making good software is very expensive.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#49

Post by gruez »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

>Well designed security models don't sell computers/operating systems, apparently.

What are you talking about? Both Android and iOS have strong sandboxing, same with mac and linux, to an extent.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#50

Post by umvi »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

> Well designed security models don't sell computers/operating systems, apparently.

Well more like it's hard to design software that is both secure-by-default and non-onerous to the end users (including devs). Every time I've tried to deploy non-trivial software systems to highly secure setups it's been a tedious nightmare. Nothing can talk to each other by default. Sometimes the filesystem is immutable and executables can't run by default. Every hole through every layer must be meticulously punched, miss one layer and things don't work and you have to trace calls through the stack, across sockets and networks, etc. to see where the holdup is. And that's not even including all the certificate/CA baggage that comes with deploying TLS-based systems.

Post reply on HN