Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

coywolf.com

Post reply on HN
Print view View original

51–60 of 116 posts

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#51

Post by ramoz »

Recall itself is absolutely ridiculous. And any solution like it is as well.

Meanwhile, Anthropic is openly pushing the ability to ingest our entire professional lives into their model which ChatGPT would happily consume as well (they're scraping up our healthcare data now).

Sandboxing is the big buzzword early 2026. I think we need to press harder for verified privacy at inference. Any data of mine or my company's going over the wire to these models needs to stay verifiably private.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#53

Post by pipo234 »

tptacek wrote: post #15

It's Signal's job to prioritize safety/privacy/security over all other concerns, and the job of an enterprise IT operation to manage risk. Underrated how different those jobs --- security and risk management --- are! Most normal people probably wouldn't enjoy working in a shop where Signal owned the risk management function, and IT/dev had to fall in line. But for the work Signal does, their near-absolutist stance ma…

That's an interesting take, but it sounds like you're downplaying the actual risks of enterprise users running agents on their desktop(?).

What would your say would be a prudent posture an IT manager should take to control risk to the organisation?

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#54

Post by tptacek »

pipo234 wrote: post #53
tptacek wrote: post #15

It's Signal's job to prioritize safety/privacy/security over all other concerns, and the job of an enterprise IT operation to manage risk. Underrated how different those jobs --- security and risk management --- are! Most normal people probably wouldn't enjoy working in a shop where Signal owned the risk management function, and IT/dev had to fall in line. But for the work Signal does, their near-absolutist stance ma…

That's an interesting take, but it sounds like you're downplaying the actual risks of enterprise users running agents on their desktop(?). What would your say would be a prudent posture an IT manager should take to control risk to the organisation?

Anybody who has ever run an internal pentest knows there's dozens of different ways to game-over an entire enterprise, and decisively resolving all of them in any organization running at scale is intractable. That's why it's called risk management, and not risk eradication.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#55

Post by mikrl »

synalx wrote: post #45
wat10000 wrote: post #43

Earlier quoted context omitted.

There are two problems that get smooshed together. One is that agents are given too much access. They need proper sandboxing. This is what you describe. The technology is there, the agents just need to use it. The other is that LLMs don't distinguish between instructions and data. This fundamentally limits what you can safely allow them to access. Seemingly simple, straightforward systems can be compromised by this.…

In that sense, AI behaves like a human assistant you hire who happens to be incredibly susceptible to social engineering.

Make sure to assign your agent all the required security trainings.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#56

Post by m4rtink »

ramoz wrote: post #51

Recall itself is absolutely ridiculous. And any solution like it is as well. Meanwhile, Anthropic is openly pushing the ability to ingest our entire professional lives into their model which ChatGPT would happily consume as well (they're scraping up our healthcare data now). Sandboxing is the big buzzword early 2026. I think we need to press harder for verified privacy at inference. Any data of mine or my company's g…

>Any data of mine or my company's going over the wire to these models needs to stay verifiably private.

I don't think this is possible without running everyting locally and the data not leaving the machine (or possibly local network) you control.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#57

Post by atoav »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

No it is also not an OS problem, it is a problem of perverse incentives.

AI companies have to monetize what they are doing. And eventually they will figure out that knowing everything about everyone can be pretty lucrative if you leverage it right and ignore or work towards abolishing existing laws that would restrict that malpractice.

There are thousand utopian worlds where LLMs knowing a lot about you could be actually a good thing. In none of them the maker of that AI has to have the prime goal of extracting as much money as possible to become the next monopolist.

Sure, the OS is one tiny technical layer users could leverage to retain some level of control. But to say this is the source of the problem is like being in a world filled with arsonists and pointing at minor fire code violations. Sure it would help to fix that, but the problem has its root entirely elsewhere.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#58

Post by pfortuny »

orbital-decay wrote: post #42
thewebguyd wrote: post #34

Earlier quoted context omitted.

> If you want the AI to do anything useful, you need to be able to trust it with the access to useful things. Sandboxing doesn't solve this. By default, AI cannot be trusted because it is not deterministic. You can't audit what the output of any given prompt is going to be to make sure its not going to rm -rf / We need some form of behavioral verification/auditing with guarantees that any input is proven to not produ…

Determinism is an absolute red herring. A correct output can be expressed in an infinite amount of ways, all of them valid. You can always make an LLM give deterministic outputs (with some overhead), that might bring you limited reproducibility, but that won't bring you correctness. You need correctness, not determinism. >We need some form of behavioral verification/auditing with guarantees that any input is proven t…

There remains the issue of responsibility, moral, technical, and legal, though.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#59

Post by SoftTalker »

m4rtink wrote: post #56
ramoz wrote: post #51

Recall itself is absolutely ridiculous. And any solution like it is as well. Meanwhile, Anthropic is openly pushing the ability to ingest our entire professional lives into their model which ChatGPT would happily consume as well (they're scraping up our healthcare data now). Sandboxing is the big buzzword early 2026. I think we need to press harder for verified privacy at inference. Any data of mine or my company's g…

>Any data of mine or my company's going over the wire to these models needs to stay verifiably private. I don't think this is possible without running everyting locally and the data not leaving the machine (or possibly local network) you control.

Once someone else knows, it's no longer a secret.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#60

Post by HiPhish »

"Hey, you know that thing no one understands how it works and has no guarantee of not going off the rails? Let's give it unrestricted access over everything!" Statements dreamed up by the utterly deranged.

I can see the value of agentic AI, but only if it has been fenced in, can only delegate actions to deterministic mechanisms, and if ever destructive decision has to be confirmed. A good example I once read about was an AI to parse customer requests: if it detects a request that the user is entitle to (e.g. cancel subscription) it will send a message like "Our AI thinks you want to cancel your subscription, is this correct?" and only after confirmation by the user will the action be carried out. To be reliable the AI itself must not determine whether the user is entitled to cancelling, it may only guess the the user's intention and then pass a message to a non-AI deterministic service. This way users don't have to wait until a human gets around to reading the message.

There is still the problem of human psychology though. If you have an AI that's 90% accurate and you have a human confirm each decision, the human's mind will start drifting off and treat 90% as if it's 100%.

Post reply on HN