Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

coywolf.com

Post reply on HN
Print view View original

31–40 of 116 posts

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#31

Post by jacquesm »

gz09 wrote: post #26
c-linkage wrote: post #23

Earlier quoted context omitted.

It's pretty clear that the security models designed into operating systems never considered networked systems. Given that most operating systems were designed and deployed before the internet, this should not be a surprise. Although one might consider it surprising that OS developers have not updated security models for this new reality, I would argue that no one wants to throw away their models due to 1) backward co…

> It's pretty clear that the security models that were design into operating systems never truly considered networked systems Andrew Tanenbaum developed the Amoeba operating system with those requirements in mind almost 40 years ago. There were plenty of others that did propose similar systems in the systems research community. It's not that we don't know how to do it just that the OS's that became mainstream didn't…

Yes, Tanenbaum was right. But it is a hard sell, even today, people just don't seem to get it.

Bluntly: if it isn't secure and correct it shouldn't be used. But companies seem to prefer insecure, incorrect but fast software because they are in competition with other parties and the ones that want to do things right get killed in the market.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#32

Post by supriyo-biswas »

MarginalGainz wrote: post #19

This resonates with what I'm seeing in the enterprise adoption layer. The pitch for 'Agentic AI' is enticing, but for mid-market operations, predictability is the primary feature, not autonomy. A system that works 90% of the time but hallucinates or leaks data the other 10% isn't an 'agent', it's a liability. We are still in the phase where 'human-in-the-loop' is a feature, not a bug.

[flagged]

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#33

Post by alphazard »

orbital-decay wrote: post #24
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

If you want the AI to do anything useful, you need to be able to trust it with the access to useful things. Sandboxing doesn't solve this. Full isolation hasn't been taken seriously because it's expensive, both in resources and complexity. Same reason why microkernels lost to monolithic ones back in the day, and why very few people use Qubes as a daily driver. Even if you're ready to pay the cost, you still need to d…

Microkernels lost "back in the day" because of how expensive syscalls were, and how many of them a microkernel requires to do basic things. That is mostly solved now, both by making syscalls faster, and also by eliminating them with things like queues in shared memory.

> you still need to design everything from the ground up

This just isn't true. The components in use now are already well designed, meaning they separate concerns well, and can be easily pulled apart. This is true of kernel code and userspace code. We just witnessed a filesystem enter and exit the linux kernel within the span of a year. No "ground up" redesign needed.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#34

Post by thewebguyd »

orbital-decay wrote: post #24
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

If you want the AI to do anything useful, you need to be able to trust it with the access to useful things. Sandboxing doesn't solve this. Full isolation hasn't been taken seriously because it's expensive, both in resources and complexity. Same reason why microkernels lost to monolithic ones back in the day, and why very few people use Qubes as a daily driver. Even if you're ready to pay the cost, you still need to d…

> If you want the AI to do anything useful, you need to be able to trust it with the access to useful things. Sandboxing doesn't solve this.

By default, AI cannot be trusted because it is not deterministic. You can't audit what the output of any given prompt is going to be to make sure its not going to rm -rf /

We need some form of behavioral verification/auditing with guarantees that any input is proven to not produce any number of specific forbidden outputs.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#35

Post by bdangubic »

alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

> AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see.

Whoever thinks/feels this has not seen enough human-written code

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#36

Post by nyrikki »

c-linkage wrote: post #23
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

It's pretty clear that the security models designed into operating systems never considered networked systems. Given that most operating systems were designed and deployed before the internet, this should not be a surprise. Although one might consider it surprising that OS developers have not updated security models for this new reality, I would argue that no one wants to throw away their models due to 1) backward co…

There is a lot to blame on the OS side, but Docker/OCI are also to blame, not allowing for permission bounds and forcing everything to the end user.

Open desktop is also problematic, but the issue is more about user land passing the buck, across multiple projects that can easily justify local decisions.

As an example, if crun set reasonable defaults and restricted namespace incompatible features by default we would be in a better position.

But docker refused to even allow you to disable the —privileged flag a decade ago,

There are a bunch of *2() system calls that decided to use caller sized structs that are problematic, and apparmor is trivial to bypass with ld_preload etc…

But when you have major projects like lamma.cpp running as container uid0, there is a lot of hardening tha could happen with projects just accepting some shared responsibility.

Containers are just frameworks to call kernel primitives, they could be made more secure by dropping more.

But OCI wants to stay simple and just stamp couple selinux/apparmor/seccomp and dbus does similar.

Berkeley sockets do force unsharing of netns etc, but Unix is about dropping privileges to its core.

Network aware is actually the easier portion, and I guess if the kernel implemented posix socket authorization it would help, but when user land isn’t even using basic features like uid/gid, no OS would work IMHO.

We need some force that incentivizes security by design and sensible defaults, right now we have wack-a-mole security theater. Strong or frozen caveman opinions win out right now.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#37

Post by ygjb »

MarginalGainz wrote: post #19

This resonates with what I'm seeing in the enterprise adoption layer. The pitch for 'Agentic AI' is enticing, but for mid-market operations, predictability is the primary feature, not autonomy. A system that works 90% of the time but hallucinates or leaks data the other 10% isn't an 'agent', it's a liability. We are still in the phase where 'human-in-the-loop' is a feature, not a bug.

> A system that works 90% of the time but hallucinates or leaks data the other 10% isn't an 'agent', it's a liability.

That strongly depends on whether or not the liability/risk to the business is internalized or externalized. Businesses take steps to mitigate internal risks while paying lip service to the risks with data and interactions where high risk is externalized. Usually that is done in the form of a waiver in the physical world, but in the digital world it's usually done through a ToS or EULA.

The big challenge is that the risks that Agentic AI in it's current incarnation or not well understood by individuals or even large businesses, and most people will happily click through thinking "I trust $vendor" to do the right thing, or "I trust my employer to prevent me doing the wrong thing."

Employers are enticed by the siren call of workforce/headcount/cost reductions and in some businesses/cases are happy to take the risk of a future realized loss as a result of an AI issue that happens after they move on/find a new role/get promoted/transfer responsibility to gain the boost of a good quarterly report.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#38

Post by jofla_net »

kuerbel wrote: post #22
apercu wrote: post #10

Earlier quoted context omitted.

I didn't mean to imply a conflict of interest, I'm wondering what product or service offering (or maybe feature on their messaging app) prompted this. No other tech (major) leaders are saying the quiet parts out loud right, about the efficacy, cost to build and operate or security and privacy nightmares created by the way we have adopted LLMs.

Maybe it's not about gaining something, but rather about not losing anything. Signal seems to operate from a kind of activism mindset, prioritizing privacy, security, and ethical responsibility, right? By warning about agentic AI, they’re not necessarily seeking a direct benefit. Or maybe the benefit is appearing more genuine and principled, which already attracted their userbase in the first place.

Exactly, if the masses cease to have "computers" any more (deterministic boxes solely under the user's control), then it matters little how bulletproof signal's ratchet protocol is, sadly.

Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

#39

Post by alphazard »

layer8 wrote: post #25
alphazard wrote: post #13

This isn't an AI problem, its an operating systems problem. AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see. Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently. That's not to say that the solution is unk…

It’s also an AI problem, because in the end we want what is called “computer use” from AI, and functionality like Recall. That’s an important part of what the CCC talk was about. The proposed solution to that is more granular, UAC-like permissions. IMO that’s not universally practical, similar to current UAC. How we can make AIs our personal assistants across our digital life — the AI effectively becoming an operatin…

We aren't there yet. You are talking about crafting a complicated window into the box holding the AI, when there isn't even a box to speak of.
Post reply on HN