Live data from Hacker News

Project Glasswing: Securing critical software for the AI era

anthropic.com

141–150 of 921 posts

Re: Project Glasswing: Securing critical software for the AI era

#141

Another Anthropic PR release based on Anthropic’s own research, uncorroborated by any outside source, where the underlying, unquestioned fact is that their model can do something incredible. > AI models have reached a level of coding capability where they can surpass all but the most skilled humans at finding and exploiting software vulnerabilities I like Anthropic, but these are becoming increasingly transparent att…

I would've basically agreed with you until I'd seen this talk: https://www.youtube.com/watch?v=1sd26pWhfmg Maybe a bad example since Nicholas works at Anthropic, but they're very accomplished and I doubt they're being misleading or even overly grandiose here See the slide 13 minutes in, which makes it look to be quite a sudden change

Very interesting, thanks for sharing.

> I doubt they're being misleading or even overly grandiose here

I think I agree.

We could definitely do much worse than Anthropic in terms of companies who can influence how these things develop.

Re: Project Glasswing: Securing critical software for the AI era

#142

The system card for Claude Mythos (PDF): https://www-cdn.anthropic.com/53566bf5440a10affd749724787c89... Interesting to see that they will not be releasing Mythos generally. [edit: Mythos Preview generally - fair to say they may release a similar model but not this exact one] I'm still reading the system card but here's a little highlight: > Early indications in the training of Claude Mythos Preview suggested that th…

If it is that dangerous as they make it appear to be, 24h does not seem sufficient time. I cannot accept this as a serious attempt.

Re: Project Glasswing: Securing critical software for the AI era

#143
BTW it seems they forgot about the part that defense uses of the model also need to be safeguarded from people. Because what if a bad person from a bad country tries to defend against peaceful attacks from a good country like the US? That would be a tragedy, so we need to limit defensive capabilities too.

Re: Project Glasswing: Securing critical software for the AI era

#144
post #76

Now, its very possible that this is Anthropic marketing puffery, but even if it is half true it still represents an incredible advancement in hunting vulnerabilities. It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobi…

You should watch this talk by Nicholas Carlini (security researcher at Anthropic). Everything in the talk was done with Opus 4.6: https://www.youtube.com/watch?v=1sd26pWhfmg

Thanks for sharing that talk, enjoyed watching it!

Re: Project Glasswing: Securing critical software for the AI era

#145

I guess we can throw out the idea that AGI is going to be democratized. In this case a sufficiently powerful model has been built and the first thing they do is only give AWS, Microsoft, Oracle ect ect access. If AGI is going to be a thing its only going to be a thing, its only going to be a thing for fortune 100 companies.. However, my guess is this is mostly the typical scare tactic marketing that Dario loves to pu…

>However, my guess is this is mostly the typical scare tactic marketing that Dario loves to push about the dangers of AI. Evaluate it yourself. Look at the exploits it discovered and decide whether you want to feel concerned that a new model was able to do that. The data is right there.

[deleted]

Re: Project Glasswing: Securing critical software for the AI era

#146

>>> the US and its allies must maintain a decisive lead in AI technology. Governments have an essential role to play in helping maintain that lead, and in both assessing and mitigating the national security risks associated with AI models. We are ready to work with local, state, and federal representatives to assist in these tasks. How long would it take to turn a defensive mechanism into an offensive one?

In this case there is almost no distinction. Assuming the model is as powerful as claimed, someone with access to the weights could do immense damage without additional significant R&D.

Which will eventually happen no matter what. That's why it's important to start preparing now.

Re: Project Glasswing: Securing critical software for the AI era

#147

Earlier quoted context omitted.

> It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobile OSes. It will likely cause some interesting tensions with government as well. eg. Apple's official stance per their 2016 customer letter is no backdoors: https://…

> If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry If Apple and Google actually cared about security of their users, they would remove a ton of obvious malware from their app stores. Instead, they tighten their walled garden pretending that it's for your security.

Some links for the downvoters:

https://news.ycombinator.com/item?id=46911901

https://news.ycombinator.com/item?id=47457963

Re: Project Glasswing: Securing critical software for the AI era

#148
post #109

I don't want to be overly cynical and am in general in favor of the contrarian attitude of simply taking people at their word, but I wonder if their current struggles with compute resources make it easier for them to choose to not deploy Mythos widely. I can imagine their safety argument is real, but regardless, they might not have the resources to profitably deploy it. (Though on the other hand, you could argue that…

I would have not believed your argument 3 months ago but I strongly suspect Anthropic actively engages in model quality throttling due to their compute constraints. Their recent deal for multi GWs worth of data center might help them correct their approach.

For what it's worth Anthropic explicity denies that. "To state it plainly: We never reduce model quality due to demand, time of day, or server load"

Also can see https://marginlab.ai/trackers/claude-code/

It's very interesting to me how widespread this conception is. Maybe it's as simple as LLM productivity degrading over time within a project, as slop compounds.

Or more recently since they added a 1m context window, maybe people are more reckless with context usage

Re: Project Glasswing: Securing critical software for the AI era

#149

Must be nice to be in a position to sell both disease and cure.

That's exactly not what they're doing. They aren't creating operating system vulnerabilities. They're telling you about ones that already existed.

Well, in a slightly indirect manner. Claude is writing a ton of code, and therefore creating a lot of security vulnerabilities.

Re: Project Glasswing: Securing critical software for the AI era

#150
Can anyone point at the critical vulnerabilities already patched as a result of mythos? (see 3:52 in the video)

For example, the 27 year old openbsd remote crash bug, or the Linux privilege escalation bugs?

I know we've had some long-standing high profile, LLM-found bugs discussed but seems unlikely there was speculation they were found by a previously unannounced frontier model.

[0] https://www.youtube.com/watch?v=INGOC6-LLv0

Post reply on HN