Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Project Glasswing: Securing critical software for the AI era

anthropic.com

Post reply on HN
Print view View original

101–110 of 921 posts

Re: Project Glasswing: Securing critical software for the AI era

#101

Post by marsven_422 »

slacktivism123 wrote: post #62
redfloatplane wrote: post #13

The system card for Claude Mythos (PDF): https://www-cdn.anthropic.com/53566bf5440a10affd749724787c89... Interesting to see that they will not be releasing Mythos generally. [edit: Mythos Preview generally - fair to say they may release a similar model but not this exact one] I'm still reading the system card but here's a little highlight: > Early indications in the training of Claude Mythos Preview suggested that th…

https://www-cdn.anthropic.com/53566bf5440a10affd749724787c89... "5.10 External assessment from a clinical psychiatrist" is a new section in this system card. Why are Anthropic like this? >We remain deeply uncertain about whether Claude has experiences or interests that matter morally, and about how to investigate or address these questions, but we believe it is increasingly important to try. We also report independen…

[dead]

Re: Project Glasswing: Securing critical software for the AI era

#102

Post by fsflover »

zachperkel wrote: post #8

Mythos Preview has already found thousands of high-severity vulnerabilities, including some in every major operating system and web browser. Scary but also cool

Every piece of software definitely has serious vulnerabilities, perfection is not achievable. Fortunately we have another approach to security: security through compartmentalization. See: https://qubes-os.org

Re: Project Glasswing: Securing critical software for the AI era

#104

Post by Analemma_ »

endunless wrote: post #15

Another Anthropic PR release based on Anthropic’s own research, uncorroborated by any outside source, where the underlying, unquestioned fact is that their model can do something incredible. > AI models have reached a level of coding capability where they can surpass all but the most skilled humans at finding and exploiting software vulnerabilities I like Anthropic, but these are becoming increasingly transparent att…

Cynicism always gets upvotes, but in this particular case, it seems fairly easy to verify if they're telling the truth? If Mythos really did find a ton of vulnerabilities, those presumably have been reported to the vendors, and are currently in the responsible nondisclosure period while they get fixed, and then after that we'll see the CVEs.

If a bunch of CVEs do in fact get published a couple months (or whatever) from now, are you going to retract this take? It's not like their claims are totally implausible: the report about Firefox security from last month was completely genuine.

Re: Project Glasswing: Securing critical software for the AI era

#105

Post by torginus »

redfloatplane wrote: post #95
torginus wrote: post #84

Earlier quoted context omitted.

This is the exact logic people that was used to claim that GPT4 was a PhD level intelligence.

You said: "I would like to reach out and talk to biologists - do you find these models to be useful and capable? Can it save you time the way a highly capable colleague would?" and they said, paraphrasing, "We reached out and talked to biologists and asked them to rank the model between 0 and 4 where 4 is a world expert, and the median people said it was a 2, which was that it helped them save time in the way a capab…

Yes that is correct. I would like a large body of experience and consenus to rely on as opposed to the regular 'trust the experts' argument, which has been shown for decades that is a deeply flawed and easy to manipulate argument.

Re: Project Glasswing: Securing critical software for the AI era

#106

Post by georgemcbay »

9cb14c1ec0 wrote: post #52

Now, its very possible that this is Anthropic marketing puffery, but even if it is half true it still represents an incredible advancement in hunting vulnerabilities. It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobi…

> It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobile OSes.

It will likely cause some interesting tensions with government as well.

eg. Apple's official stance per their 2016 customer letter is no backdoors:

https://www.apple.com/customer-letter/

Will they be allowed to maintain that stance in a world where all the non-intentional backdoors are closed? The reason the FBI backed off in 2016 is because they realized they didn't need Apple's help:

https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_d...

What happens when that is no longer true, especially in today's political climate?

Re: Project Glasswing: Securing critical software for the AI era

#107

Post by dakolli »

I guess we can throw out the idea that AGI is going to be democratized. In this case a sufficiently powerful model has been built and the first thing they do is only give AWS, Microsoft, Oracle ect ect access.

If AGI is going to be a thing its only going to be a thing, its only going to be a thing for fortune 100 companies..

However, my guess is this is mostly the typical scare tactic marketing that Dario loves to push about the dangers of AI.

Re: Project Glasswing: Securing critical software for the AI era

#108

Post by bonsai_spool »

torginus wrote: post #96
bonsai_spool wrote: post #70

Earlier quoted context omitted.

> Just reading this, the inevitable scaremongering about biological weapons comes up. It's very easy to learn more about this if it's seriously a question you have. I don't quite follow why you think that you are so much more thoughtful than Anthropic/OpenAI/Google such that you agree that LLMs can't autonomously create very bad things but—in this area that is not your domain of expertise—you disagree and insist that…

>It's very easy to learn more about this if it's seriously a question you have. No, it's not. It took years of polishing by software engineers, who understand this exact profession to get models where they are now. Despite that, most engineers were of the opinion, that these models were kinda mid at coding, up until recently, despite these models far outperforming humans in stuff like competitive programming. Yet des…

>>It's very easy to learn more about this if it's seriously a question you have.

>No, it's not. It took years of polishing by software engineers, who understand this exact profession to get models where they are now

This reads as defensive. The thing that is easy to learn is 'why are biology ai LLMs dangerous chatgpt claude'. I have never googled this before, so I'll do this with the reader, live. I'm applying a date cutoff of 12/31/24 by the way.

Here, dear reader, are the first five links. I wish I were lying about this:

- https://sciencebusiness.net/news/ai/scientists-grapple-risk-...

- https://www.governance.ai/analysis/managing-risks-from-ai-en...

- https://gssr.georgetown.edu/the-forum/topics/biosec/the-doub...

- https://www.vox.com/future-perfect/23820331/chatgpt-bioterro...

- https://www.reddit.com/r/ClaudeAI/comments/1de8qkv/awareness...

I don't know about you, but that counts as easy to me.

-----

> I would apply this framework to biology - this time, expert effort, and millions of GPU hours and a giant corpus that is open source clearly has not been involved in biology.

I've been getting good programming and molecular biology results out of these back to GPT3.5.

I don't know what to tell you—if you really wanted to understand the importance, you'd know already.

Re: Project Glasswing: Securing critical software for the AI era

#109

Post by Sol- »

I don't want to be overly cynical and am in general in favor of the contrarian attitude of simply taking people at their word, but I wonder if their current struggles with compute resources make it easier for them to choose to not deploy Mythos widely. I can imagine their safety argument is real, but regardless, they might not have the resources to profitably deploy it. (Though on the other hand, you could argue that they could always simply charge more.)

Re: Project Glasswing: Securing critical software for the AI era

#110

Post by fsflover »

justincormack wrote: post #73
pants2 wrote: post #59

Earlier quoted context omitted.

Software security heavily favors the defenders (ex. it's much easier to encrypt a file than break the encryption). Thus with better tools and ample time to reach steady-state, we would expect software to become more secure.

Software security heavily favours the attacker (ex. its much easier to find a single vulnerability than to patch every vulnerability). Thus with better tools and ample time to reach steady-state, we would expect software to remain insecure.

This is only true if your approach is security through correctness. This never works in practice. Try security through compartmentalization. Qubes OS provides it reasonably well.
Post reply on HN