Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

coywolf.com

Post reply on HN
Print view View original

21–30 of 116 posts

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#21

Post by JoshTriplett »

apercu wrote: post #2

A large percentage of my work is peripheral to info security (ISO 27001, CMMC, SOC 2), and I've been building internet companies and software since the 90's (so I have a technical background as well), which makes me think that I'm qualified to have an opinion here. And I completely agree that LLMs (the way they have been rolled out for most companies, and how I've witnessed them being used) are an incredibly underest…

Signal is conveying a message of wanting to be able to trust your technology/tools to work for you and work reliably. This is a completely reasonable message, and it's the best kind of healthy messaging: "apply this objectively good standard, and you will find that you want to use tools like ours".

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#22

Post by kuerbel »

apercu wrote: post #10
jsheard wrote: post #5

Earlier quoted context omitted.

> But on the other hand, I'm pragmatic (some might say cynical?), and I'm just left here thinking "what is Signal trying to sell us?" A messaging app? I'm struggling to come up with a potential conflict of interest here unless they have a wild pivot coming up.

I didn't mean to imply a conflict of interest, I'm wondering what product or service offering (or maybe feature on their messaging app) prompted this. No other tech (major) leaders are saying the quiet parts out loud right, about the efficacy, cost to build and operate or security and privacy nightmares created by the way we have adopted LLMs.

Maybe it's not about gaining something, but rather about not losing anything. Signal seems to operate from a kind of activism mindset, prioritizing privacy, security, and ethical responsibility, right? By warning about agentic AI, they’re not necessarily seeking a direct benefit. Or maybe the benefit is appearing more genuine and principled, which already attracted their userbase in the first place.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#23

Post by c-linkage »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

It's pretty clear that the security models designed into operating systems never considered networked systems. Given that most operating systems were designed and deployed before the internet, this should not be a surprise.

Although one might consider it surprising that OS developers have not updated security models for this new reality, I would argue that no one wants to throw away their models due to 1) backward compatibility; and 2) the amount of work it would take to develop and market an entirely new operating system that is fully network aware.

Yes we have containers and VMs, but these are just kludges on top of existing systems to handle networks and tainted (in the Perl sense) data.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#24

Post by orbital-decay »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

If you want the AI to do anything useful, you need to be able to trust it with the access to useful things. Sandboxing doesn't solve this.

Full isolation hasn't been taken seriously because it's expensive, both in resources and complexity. Same reason why microkernels lost to monolithic ones back in the day, and why very few people use Qubes as a daily driver. Even if you're ready to pay the cost, you still need to design everything from the ground up, or at least introduce low attack surface interfaces, which still leads to pretty major changes to existing ecosystems.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#25

Post by layer8 »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

It’s also an AI problem, because in the end we want what is called “computer use” from AI, and functionality like Recall. That’s an important part of what the CCC talk was about. The proposed solution to that is more granular, UAC-like permissions. IMO that’s not universally practical, similar to current UAC. How we can make AIs our personal assistants across our digital life — the AI effectively becoming an operating system from the user’s point of view — with security and reliability, is a hard problem.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#26

Post by gz09 »

c-linkage wrote: post #23
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

It's pretty clear that the security models designed into operating systems never considered networked systems. Given that most operating systems were designed and deployed before the internet, this should not be a surprise. Although one might consider it surprising that OS developers have not updated security models for this new reality, I would argue that no one wants to throw away their models due to 1) backward co…

> It's pretty clear that the security models that were design into operating systems never truly considered networked systems

Andrew Tanenbaum developed the Amoeba operating system with those requirements in mind almost 40 years ago. There were plenty of others that did propose similar systems in the systems research community. It's not that we don't know how to do it just that the OS's that became mainstream didn't want to/need to/consider those requirements necessary/.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#27

Post by HPsquared »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

Android servers? They already have ARM servers.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#28

Post by OptionOfT »

c-linkage wrote: post #23
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

It's pretty clear that the security models designed into operating systems never considered networked systems. Given that most operating systems were designed and deployed before the internet, this should not be a surprise. Although one might consider it surprising that OS developers have not updated security models for this new reality, I would argue that no one wants to throw away their models due to 1) backward co…

> It's pretty clear that the security models designed into operating systems never considered networked systems. Given that most operating systems were designed and deployed before the internet, this should not be a surprise.

I think Active Directory comes pretty close. I remember the days where we had an ASP.NET application where we signed in with our Kerberos credentials, which flowed to the application, and the ASP.NET app connected to MSSQL using my delegated credentials.

When the app then uploaded my file to a drive, it was done with my credentials, if I didn't have permission it would fail.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#29

Post by api »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

> Well designed security models don't sell computers/operating systems, apparently.

That's because there's a tension between usability and security, and usability sells. It's possible to engineer security systems that minimize this, but that is extremely hard and requires teams of both UI/UX people and security experts or people with both skill sets.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#30

Post by witnessme »

MarginalGainz wrote: post #19

This resonates with what I'm seeing in the enterprise adoption layer. The pitch for 'Agentic AI' is enticing, but for mid-market operations, predictability is the primary feature, not autonomy. A system that works 90% of the time but hallucinates or leaks data the other 10% isn't an 'agent', it's a liability. We are still in the phase where 'human-in-the-loop' is a feature, not a bug.

Can't agree more
Post reply on HN