Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

coywolf.com

Post reply on HN
Print view View original

111–116 of 116 posts

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#111

Post by captn3m0 »

theshrike79 wrote: post #110
umvi wrote: post #50

Earlier quoted context omitted.

> Well designed security models don't sell computers/operating systems, apparently. Well more like it's hard to design software that is both secure-by-default and non-onerous to the end users (including devs). Every time I've tried to deploy non-trivial software systems to highly secure setups it's been a tedious nightmare. Nothing can talk to each other by default. Sometimes the filesystem is immutable and executabl…

There's a reason why all security professionals I know use an iPhone. To my knowledge there hasn't been a single case of an iOS application being able to read the data of another application - or OS files it wasn't explicitly given authorisation to do so. It can be done, but for desktop it has never been a priority. A bit like the earliest versions of Windows encountering The Internet for the first time. They were bu…

Lots of sandbox escapes on iOS, but my favorite was https://blog.siguza.net/psychicpaper/

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#112

Post by fsflover »

umvi wrote: post #50
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

> Well designed security models don't sell computers/operating systems, apparently. Well more like it's hard to design software that is both secure-by-default and non-onerous to the end users (including devs). Every time I've tried to deploy non-trivial software systems to highly secure setups it's been a tedious nightmare. Nothing can talk to each other by default. Sometimes the filesystem is immutable and executabl…

> Well more like it's hard to design software that is both secure-by-default and non-onerous to the end users (including devs).

Doesn't Qubes OS count?

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#113

Post by tremon »

c-linkage wrote: post #23
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

It's pretty clear that the security models designed into operating systems never considered networked systems. Given that most operating systems were designed and deployed before the internet, this should not be a surprise. Although one might consider it surprising that OS developers have not updated security models for this new reality, I would argue that no one wants to throw away their models due to 1) backward co…

the security models designed into operating systems never considered networked systems

The security model was aimed at putting the user in control of the software they run. That's what general-purpose computing is: allowing the user to use the machine's resources for whatever general purpose they intend. The only protection required was to make sure the user couldn't interfere with other users on the same system.

What was never considered before is adversarial software. The model we're now operating under is that users are no longer in control of the software they run. That is the primary thing that has changed; not the users, not the network, but the provenance and accountability of software.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#114

Post by alphazard »

lucketone wrote: post #109
alphazard wrote: post #95

Earlier quoted context omitted.

> Every time I've tried to deploy non-trivial software systems to highly secure setups it's been a tedious nightmare. I don't know exactly which "secure setups" you are talking about, but the false equivalency between security and complexity is mostly from security theater. If you start with insecure systems and then do extra things to make them secure, then that additional complexity interacts with the thing you are…

Whats wrong with firewalls? Or, how the alternative world looks where network security is more pleasant?

Firewalls are a fundamentally bad approach and are avoidable with good design.

Nothing should have access to the network by default. You can either get that right by limiting resource access (which is the job of the operating system) or you can get it wrong and have to expose new APIs and hooks to invite an ecosystem of many, slightly different, complicated tools to configure network access.

To give access to the network, you spawn the process with a handle to the port it can listen on, or a handle to a dynamically allocated port that it can only dial out of. This is no more complicated than configuration, and it doesn't have to be difficult for users. It can bubble up to a GUI very similar to what the iPhone has for giving access to location, contacts list, etc.

The fact that most "security" people have a knee-jerk reaction to "firewall bad" is exactly the cultural problem that I'm talking about. It's not a technical problem anymore, the solutions are known, but they aren't widely known, and they aren't known by decision makers. We've become so used to the wrong way for so long that highly trained people reliably have bad taste.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#115

Post by benban »

the point about this being an os problem not an ai problem resonates. letting untrusted agents drive your browser smells like a problem to me.

in practice we've had better luck running agents in lightweight sandboxes with explicit capability handles. curious if anyone's tried capability-based systems like sel4 for hosting agents, feels like mainstream oses have a long way to go here.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#116

Post by m3047 »

m3047 wrote: post #94
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

In exasperation, people truly concerned about security / secops are turning to unikernels and shell-free OS; at the same time agents are all in on curl | bash and other cheap hacks.

Come on, really? Do you think April Fools came early? https://platform.claude.com/docs/en/agents-and-tools/tool-us...
Post reply on HN