Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Project Glasswing: Securing critical software for the AI era

anthropic.com

Post reply on HN
Print view View original

131–140 of 921 posts

Re: Project Glasswing: Securing critical software for the AI era

#131

Post by TheAtomic »

redfloatplane wrote: post #77
slacktivism123 wrote: post #62

Earlier quoted context omitted.

https://www-cdn.anthropic.com/53566bf5440a10affd749724787c89... "5.10 External assessment from a clinical psychiatrist" is a new section in this system card. Why are Anthropic like this? >We remain deeply uncertain about whether Claude has experiences or interests that matter morally, and about how to investigate or address these questions, but we believe it is increasingly important to try. We also report independen…

A thought experiment: It's April, 1991. Magically, some interface to Claude materialises in London. Do you think most people would think it was a sentient life form? How much do you think the interface matters - what if it looks like an android, or like a horse, or like a large bug, or a keyboard on wheels? I don't come down particularly hard on either side of the model sapience discussion, but I don't think dismissi…

Isn't this the premise of Garfield's Ex Machina?

Re: Project Glasswing: Securing critical software for the AI era

#132

Post by tptacek »

pants2 wrote: post #59
jryio wrote: post #11

Let's fast forward the clock. Does software security converge on a world with fewer vulnerabilities or more? I'm not sure it converges equally in all places. My understanding is that the pre-AI distribution of software quality (and vulnerabilities) will be massively exaggerated. More small vulnerable projects and fewer large vulnerable ones. It seems that large technology and infrastructure companies will be able to…

Software security heavily favors the defenders (ex. it's much easier to encrypt a file than break the encryption). Thus with better tools and ample time to reach steady-state, we would expect software to become more secure.

I don't think this is broadly true and to the extent it's true for cryptographic software, it's only relatively recently become true; in the 2000s and 2010s, if I was tasked with assessing software that "encrypted a file" (or more likely some kind of "message"), my bet would be on finding a game-over flaw in that.

Re: Project Glasswing: Securing critical software for the AI era

#133

Post by tptacek »

torginus wrote: post #81
SheinhardtWigCo wrote: post #56

Society is about to pay a steep price for the software industry's cavalier attitude toward memory safety and control flow integrity.

Thank god, finally someone said it. I don't know the first thing about cybersecurity, but in my experience all these sandbox-break RCEs involve a step of highjacking the control flow. There were attempts to prevent various flavors of this, but imo, as long as dynamic branches exist in some form, like dlsym(), function pointers, or vtables, we will not be rid of this class of exploit entirely. The latter one is the mo…

Everybody has been saying this for the last 15 years.

Re: Project Glasswing: Securing critical software for the AI era

#135

Post by timschmidt »

torginus wrote: post #64
timschmidt wrote: post #34

Earlier quoted context omitted.

Most vulnerabilities seem to be in C/C++ code, or web things like XSS, unsanitized input, leaky APIs, etc. Perhaps a chunk of that token spend will be porting legacy codebases to memory safe languages. And fewer tokens will be required to maintain the improved security.

I think most vulnerabilities are in crappy enterprise software. TOCTOU stuff in the crappy microservice cloud app handling patient records at your hospital, shitty auth at a webshop, that sort of stuff. A lot of these stuff is vulnerable by design - customer wanted a feature, but engineering couldnt make it work securely with the current architecture - so they opened a tiny hole here and there, hopefully nobody will…

Yeah. Crufty cobbled together enterprise stuff will suffer some of the worst. But this will be a great opportunity for the enterprise software services economy! lol.

I honestly see some sort of automated whole codebase auditing and refactoring being the next big milestone along the chatbot -> claude code / codex / aider -> multi-agent frameworks line of development. If one of the big AI corps cracks that problem then all this goes away with the click of a button and exchange of some silver.

Re: Project Glasswing: Securing critical software for the AI era

#136

Post by endunless »

Analemma_ wrote: post #104
endunless wrote: post #15

Another Anthropic PR release based on Anthropic’s own research, uncorroborated by any outside source, where the underlying, unquestioned fact is that their model can do something incredible. > AI models have reached a level of coding capability where they can surpass all but the most skilled humans at finding and exploiting software vulnerabilities I like Anthropic, but these are becoming increasingly transparent att…

Cynicism always gets upvotes, but in this particular case, it seems fairly easy to verify if they're telling the truth? If Mythos really did find a ton of vulnerabilities, those presumably have been reported to the vendors, and are currently in the responsible nondisclosure period while they get fixed, and then after that we'll see the CVEs. If a bunch of CVEs do in fact get published a couple months (or whatever) fr…

> If a bunch of CVEs do in fact get published a couple months (or whatever) from now, are you going to retract this take?

I would like to think that I would, yes.

What it comes down to, for me, is that lately I have been finding that when Anthropic publishes something like this article – another recent example is the AI and emotions one – if I ask the question, does this make their product look exceptionally good, especially to a casual observer just scanning the headlines or the summary, the answer is usually yes.

This feels especially true if the article tries to downplay that fact (they’re not _real_ emotions!) or is overall neutral to negative about AI in general, like this Glasswing one (AI can be a security threat!).

Re: Project Glasswing: Securing critical software for the AI era

#137

Post by supern0va »

dakolli wrote: post #107

I guess we can throw out the idea that AGI is going to be democratized. In this case a sufficiently powerful model has been built and the first thing they do is only give AWS, Microsoft, Oracle ect ect access. If AGI is going to be a thing its only going to be a thing, its only going to be a thing for fortune 100 companies.. However, my guess is this is mostly the typical scare tactic marketing that Dario loves to pu…

>However, my guess is this is mostly the typical scare tactic marketing that Dario loves to push about the dangers of AI.

Evaluate it yourself. Look at the exploits it discovered and decide whether you want to feel concerned that a new model was able to do that. The data is right there.

Re: Project Glasswing: Securing critical software for the AI era

#139

Post by zb3 »

> On the global stage, state-sponsored attacks from actors like China, Iran, North Korea, and Russia have threatened to compromise the infrastructure that underpins both civilian life and military readiness.

Yeah, makes sense. Those countries are bad because they execute state-sponsored cyber attacks, the US and Israel on the other hand are good, they only execute state-sponsored defense.

Re: Project Glasswing: Securing critical software for the AI era

#140

Post by zb3 »

Miraste wrote: post #27

>We plan to launch new safeguards with an upcoming Claude Opus model, allowing us to improve and refine them with a model that does not pose the same level of risk as Mythos Preview2. This seems like the real news. Are they saying they're going to release an intentionally degraded model as the next Opus? Big opportunity for the other labs, if that's true.

Well since Anthropic treats us as second class evil citizens, I guess they don't want our evil money either.
Post reply on HN