Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Project Glasswing: Securing critical software for the AI era

anthropic.com

Post reply on HN
Print view View original

321–330 of 921 posts

Re: Project Glasswing: Securing critical software for the AI era

#321

Post by charcircuit »

kristofferR wrote: post #286

This is pretty insane. A model so powerful they felt that releasing it would create a netsec tsunami if released publicly. AGI isn't here yet, but we don't need to get there for massive societal effects. How long will they hold off, especially as competitors are getting closer to their releases of equally powerful models?

OpenAI did the same thing with GPT3 trying to scare people into thinking it would end the internet. OpenAI even reached out to someone who reproduced a weaker version of GPT3 and convinced him to change his mind about releasing it publicly due to how much "harm" it would cause.

These claims of how much harm the models will cause is always overblown.

Re: Project Glasswing: Securing critical software for the AI era

#322

Post by Ar-Curunir »

realo wrote: post #307
simonsarris wrote: post #282

Earlier quoted context omitted.

Presumably, those have influenced elections, though I guess it depends what you count as an attack. Plenty of bots try to modify public opinion. Someone hacked the DNC in 2015/16, the result of which also alleged attempted manipulation in 2008: https://en.wikipedia.org/wiki/Democratic_National_Committee_... Since we (as old Rummy said) do not know what we do not know, we cannot be certain about the extent of cyber at…

Yes... they might have influenced elections and now, as a result, the world must cope with the Trump regime. Let's now fool ourselves.... Trump is probably the best, most successful attempt at world de-stabilisation all those rogue states ever achieved.

Maybe Americans should take responsibility for electing a maniac as their President. In the end, the buck stops with Americans.

Re: Project Glasswing: Securing critical software for the AI era

#323

Post by devmor »

kranke155 wrote: post #278
atlgator wrote: post #266

[flagged]

It’s insane. This is what - could we say it’s beyond AGI at least in cybersecurity? This is a real wake up call. On some of this stuff, the AI’s “uneven intelligence” is becoming absurdly high at its local peaks.

Please stop using terms you don’t understand like “AGI” because you feel overwhelmed by something doing cool stuff. It’s exhausting.

Re: Project Glasswing: Securing critical software for the AI era

#324

Post by tptacek »

stephc_int13 wrote: post #291
tptacek wrote: post #223

Earlier quoted context omitted.

I don't think it matters one way or the other to your thesis but I'm skeptical that state-level CNE organizations were hoarding vulnerabilities before; my understanding is that at least on the NATO side of the board they were all basically carefully managing an enablement pipeline that would have put them N deep into reliable exploit packages, for some surprisingly small N. There are a bunch of little reasons why the…

The economics would be different in say, North Korea, don't you think?

Why? What do you mean?

Re: Project Glasswing: Securing critical software for the AI era

#325

Post by socketcluster »

doug_durham wrote: post #275
SheinhardtWigCo wrote: post #56

Society is about to pay a steep price for the software industry's cavalier attitude toward memory safety and control flow integrity.

I think society is going to start paying the price for humans being human. As the paper points out there is a lot of good faith, serious software that has vulnerabilities. These aren't projects you would characterize as people being cavalier. It is simply beyond the limits of humans to create vulnerability-free software of high complexity. That's why high reliability software depends on extreme simplicity and strict…

100%, poorly architected software is really difficult to make secure. I think this will extend to AI as well. It will just dial up the complexity of the code until bugs and vulnerabilities start creeping in.

At some point, people will have to decide to stop the complexity creep and try to produce minimal software.

For any complex project with 100k+ lines of code, the probability that it has some vulnerabilities is very high. It doesn't fit into LLM context windows and there aren't enough attention heads to attend to every relevant part. On the other hand, for a codebase which is under 1000 lines, you can be much more confident that the LLM didn't miss anything.

Also, the approach of feeding the entire codebase to an LLM in parts isn't going to work reliably because vulnerabilities often involve interactions between different parts of the code. Both parts of the code may look fine if considered independently but together they create a vulnerability.

Good architecture is critical now because you really need to be able to have the entire relevant context inside the LLM context window... When considering the totality of all software, this can only be achieved through an architecture which adheres to high cohesion and loose coupling principles.

Re: Project Glasswing: Securing critical software for the AI era

#326

Post by evanmoran »

pugworthy wrote: post #169
baddash wrote: post #58

> security product > glass in the name

I had a team mate propose a new security layer for an industrial device which he wanted to call "Eggshell"

We shall call it Achilles, as Claude Mythos is its only weakness.

Re: Project Glasswing: Securing critical software for the AI era

#327

Post by spr-alex »

georgemcbay wrote: post #106
9cb14c1ec0 wrote: post #52

Now, its very possible that this is Anthropic marketing puffery, but even if it is half true it still represents an incredible advancement in hunting vulnerabilities. It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobi…

> It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobile OSes. It will likely cause some interesting tensions with government as well. eg. Apple's official stance per their 2016 customer letter is no backdoors: https://…

Adding to your comment a similar letter was published as recently as September 2025 https://support.apple.com/en-us/122234 "we have never built a backdoor or master key to any of our products or services and we never will."

Re: Project Glasswing: Securing critical software for the AI era

#328

Post by mcast »

Gigachad wrote: post #243
9cb14c1ec0 wrote: post #52

Now, its very possible that this is Anthropic marketing puffery, but even if it is half true it still represents an incredible advancement in hunting vulnerabilities. It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobi…

Apple has already largely crushed hacking with memory tagging on the iPhone 17 and lockdown mode. Architectural changes, safer languages, and sandboxing have done more for security than just fixing bugs when you find them.

Lockdown mode is opt-in only though

Re: Project Glasswing: Securing critical software for the AI era

#329

Post by jeffmcjunkin »

fintech_eng wrote: post #189
woeirua wrote: post #76

Earlier quoted context omitted.

You should watch this talk by Nicholas Carlini (security researcher at Anthropic). Everything in the talk was done with Opus 4.6: https://www.youtube.com/watch?v=1sd26pWhfmg

its also very easy to reproduce. i have more findings than i know what to do with

Can confirm.

Re: Project Glasswing: Securing critical software for the AI era

#330

Post by rlopc »

atlgator wrote: post #266

[flagged]

These issues are always found in the same kinds of projects that support an insane amount of largely unused protocols and features like ffmpeg, sudo, curl.

OpenBSD has many unexplored corners and also (irresponsibly IMO) maintains forks of other projects in base.

A motivated human could find all of these probably by writing 100% code coverage and fuzzing.

The market for these tools is very small. Good luck applying them to a release of sqlite or postfix.

I don't understand how people here are hyping this up, unless they work for AI related companies as probably 80% of them do. People have found these issues for decades without AI. Sure, you can generate fuzzing code and find one or two issues in the usual suspects. Better do it manually and understand your own code.

Post reply on HN