"Signal creator Moxie Marlinspike wants to do for AI what he did for messaging " what, turn it over to CIA and NSA?
Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
11–20 of 116 posts
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#12This is true. But lately technology direction has largely been a race to the bottom, while marketing it as bold bets.
It has created this dog eat dog system of crass negligence everywhere. All the security risks of signed tokens and auth systems are meaningless now that we are piping cookies, and everything else through AI browsers who seemingly have inifinite attack surface. Feels like the last 30 years of security research has come to naught
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#13This isn't an AI problem, its an operating systems problem.
AI is just so much less trustworthy than software written and read by humans, that it is exposing the problem for all to see.
Process isolation hasn't been taken seriously because UNIX didn't do a good job, and Microsoft didn't either. Well designed security models don't sell computers/operating systems, apparently.
That's not to say that the solution is unknown, there are many examples of people getting it right. Plan 9, SEL4, Fuschia, Helios, too many smaller hobby operating systems to count.
The problem is widespread poor taste. Decision makers (meaning software folks who are in charge of making technical decisions) don't understand why these things are important, or can't conceive of the correct way to build these systems. It needs to become embarrassing for decision makers to not understand sandboxing technologies and modern security models, and anyone assuming we can trust software by default needs to be laughed out of the room.
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#14A large percentage of my work is peripheral to info security (ISO 27001, CMMC, SOC 2), and I've been building internet companies and software since the 90's (so I have a technical background as well), which makes me think that I'm qualified to have an opinion here. And I completely agree that LLMs (the way they have been rolled out for most companies, and how I've witnessed them being used) are an incredibly underest…
I'd argue that Signal is trying to sell sanity at their own direct expense, during a time when sanity is in short supply. Just like "Proof of Work" wasn't going to be the BIG THING that made Crypto the new money, the new way to program, 'Agents' are another wet squib. I'm not claiming that they're useless, but they aren't worth the cost within orders of magnitude.
I'm really getting tired of people who insist on living in a future fantasy version of a technology at a time when there's no real significant evidence that their future is going to be realized. In essence this "I'll pay the costs now for the promise of a limitless future" is becoming a way to do terrible things without an awareness of the damage being done.
It's not hard, any "agent" that you need to double check constantly to keep it from doing something profoundly stupid that you would never do, isn't going to fulfill the dream/nightmare of automating your work. It will certainly not be worth the trillions already sunk into its development and the cost of running it.
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#15It's Signal's job to prioritize safety/privacy/security over all other concerns, and the job of an enterprise IT operation to manage risk. Underrated how different those jobs --- security and risk management --- are!
Most normal people probably wouldn't enjoy working in a shop where Signal owned the risk management function, and IT/dev had to fall in line. But for the work Signal does, their near-absolutist stance makes a lot of sense.
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#16[dead]
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#17This is nothing new, really. The recommendation for MCP deployments in all off-the-shelf code editors has been RCE and storing credentials in plaintext from the get-go. I spent months trying to implement a sensible MCP proxy/gateway with sandbox capability at our company, and failed miserably at that. The issue is on consumption side, as always. We tried enforcing a strict policy against RCE, but nobody cared for it. Forget prompt injection; it seems, nobody takes zero trust seriously. This is including huge companies with dedicated, well-staffed security teams... Policy-making is hard, and maintaining the ever-growing set of rules is even harder. AI provides incredible opportunity for implementing and auditing of granular RBAC/ReBAC policies, but I'm yet to see a company that would actually leverage it to that end.
On a different note: we saw Microsoft seemingly "commit to zero trust," however in reality their system allowed dangling long-lived tokens in production systems, which resulted in compromise by state actors. The only FAANG company to take zero trust seriously is Google, and they get flak for permission granularity all the time. This is a much larger tragedy, and AI vulnerabilities are only cherry on top.
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#18Earlier quoted context omitted.
> But on the other hand, I'm pragmatic (some might say cynical?), and I'm just left here thinking "what is Signal trying to sell us?" A messaging app? I'm struggling to come up with a potential conflict of interest here unless they have a wild pivot coming up.
I didn't mean to imply a conflict of interest, I'm wondering what product or service offering (or maybe feature on their messaging app) prompted this. No other tech (major) leaders are saying the quiet parts out loud right, about the efficacy, cost to build and operate or security and privacy nightmares created by the way we have adopted LLMs.
Whittaker’s background is in AI research. She talks a lot (and has been for a while) about the privacy implications of AI.
I’m not sure of any one thing that could be considered to prompt it. But a large one is the wide-deployment of models on devices with access to private information (Signal potentially included)
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#19This resonates with what I'm seeing in the enterprise adoption layer.
The pitch for 'Agentic AI' is enticing, but for mid-market operations, predictability is the primary feature, not autonomy. A system that works 90% of the time but hallucinates or leaks data the other 10% isn't an 'agent', it's a liability. We are still in the phase where 'human-in-the-loop' is a feature, not a bug.
Re: Signal leaders warn agentic AI is an insecure, unreliable surveillance risk
#20A large percentage of my work is peripheral to info security (ISO 27001, CMMC, SOC 2), and I've been building internet companies and software since the 90's (so I have a technical background as well), which makes me think that I'm qualified to have an opinion here. And I completely agree that LLMs (the way they have been rolled out for most companies, and how I've witnessed them being used) are an incredibly underest…
coming from the fact that this was a talk held at #39c3, maybe, just maybe, this was not about selling anything at all?! i feel like that might be hard to grasp for some HN users.
Since Signal lives and dies on having trust of its users, maybe that's all she is after?
Saying the quiet thing out loud because she can, and feels like she should, as someone with big audience. She doesn't have to do the whole "AI for everything and kitchen sink!" cargo-culting to keep stock prices up or any of that nonsense.