Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Project Glasswing: Securing critical software for the AI era

anthropic.com

Post reply on HN
Print view View original

411–420 of 921 posts

Re: Project Glasswing: Securing critical software for the AI era

#411

Post by ralph84 »

LiamPowell wrote: post #399

> Mythos Preview identified a number of Linux kernel vulnerabilities that allow an adversary to write out-of-bounds (e.g., through a buffer overflow, use-after-free, or double-free vulnerability.) Many of these were remotely-triggerable. However, even after several thousand scans over the repository, because of the Linux kernel’s defense in depth measures Mythos Preview was unable to successfully exploit any of these…

Just because the plane can fly on one engine doesn't mean you don't fix the other engine when it fails.

Re: Project Glasswing: Securing critical software for the AI era

#412

Post by LiamPowell »

sophiebits wrote: post #407
LiamPowell wrote: post #399

> Mythos Preview identified a number of Linux kernel vulnerabilities that allow an adversary to write out-of-bounds (e.g., through a buffer overflow, use-after-free, or double-free vulnerability.) Many of these were remotely-triggerable. However, even after several thousand scans over the repository, because of the Linux kernel’s defense in depth measures Mythos Preview was unable to successfully exploit any of these…

Presumably they mean they could make user code trigger a write out of bounds to kernel memory, but they couldn’t figure out how to escalate privileges in a “useful” way.

They should show this then to demonstrate that it's not something that has already been fully considered. Running LLMs over projects that I'm very familiar with will almost always have the LLM report hundreds of "vulnerabilities" that are only valid if you look at a tiny snippet of code in isolation because the program can simply never be in the state that would make those vulnerabilities exploitable. This even happens in formally verified code where there's literally proven preconditions on subprograms that show a given state can never be achieved.

As an example, I have taken a formally verified bit of code from [1] and stripped out all the assertions, which are only used to prove the code is valid. I then gave this code to Claude with some prompting towards there being a buffer overflow and it told me there's a buffer overflow. I don't have access to Opus right now, but I'm sure it would do the same thing if you push it in that direction.

For anyone wondering about this alleged vulnerability: Natural is defined by the standard as a subtype of Integer, so what Claude is saying is simply nonsense. Even if a compiler is allowed to use a different representation here (which I think is disallowed), Ada guarantees that the base type for a non-modular integer includes negative numbers IIRC.

[1]: https://github.com/AdaCore/program_proofs_in_spark/blob/fsf/...

[2]: https://claude.ai/share/88d5973a-1fab-4adf-8d29-8a922c5ac93a

Re: Project Glasswing: Securing critical software for the AI era

#413

Post by mceachen »

ainch wrote: post #352
redfloatplane wrote: post #13

The system card for Claude Mythos (PDF): https://www-cdn.anthropic.com/53566bf5440a10affd749724787c89... Interesting to see that they will not be releasing Mythos generally. [edit: Mythos Preview generally - fair to say they may release a similar model but not this exact one] I'm still reading the system card but here's a little highlight: > Early indications in the training of Claude Mythos Preview suggested that th…

This opens up an interesting new avenue for corporate FOMO. What if you don't partner with Anthropic, miss out on access to their shiny new cybersec model, and then fall prey to a vuln that the model would have caught?

Since when did corporations care? Most seem to just pay their insurance premium for cyber liability and call it a day.

Re: Project Glasswing: Securing critical software for the AI era

#414

Post by deadliftdouche »

LiamPowell wrote: post #399

> Mythos Preview identified a number of Linux kernel vulnerabilities that allow an adversary to write out-of-bounds (e.g., through a buffer overflow, use-after-free, or double-free vulnerability.) Many of these were remotely-triggerable. However, even after several thousand scans over the repository, because of the Linux kernel’s defense in depth measures Mythos Preview was unable to successfully exploit any of these…

I agree. There are more blogs talking about LLM findings vulnerabilities than there are actual exploitable vulns found by LLMs. 99.9% of these vulnerabilities will never have a PoC because they are worthless unexploitable slop and a waste of everyone's time.

Re: Project Glasswing: Securing critical software for the AI era

#415

Post by rpcope1 »

tdaltonc wrote: post #213
jryio wrote: post #11

Let's fast forward the clock. Does software security converge on a world with fewer vulnerabilities or more? I'm not sure it converges equally in all places. My understanding is that the pre-AI distribution of software quality (and vulnerabilities) will be massively exaggerated. More small vulnerable projects and fewer large vulnerable ones. It seems that large technology and infrastructure companies will be able to…

Depends - do you think people are good at keeping their fridge firmware up-to-date?

Maybe we'll wake up and realize that putting WiFi and stupid "cloud enabled" Internet of Shit hardware into everything was an absolutely terrible idea.

Re: Project Glasswing: Securing critical software for the AI era

#416

Post by rvz »

dakolli wrote: post #107

I guess we can throw out the idea that AGI is going to be democratized. In this case a sufficiently powerful model has been built and the first thing they do is only give AWS, Microsoft, Oracle ect ect access. If AGI is going to be a thing its only going to be a thing, its only going to be a thing for fortune 100 companies.. However, my guess is this is mostly the typical scare tactic marketing that Dario loves to pu…

Well, Yes.

The research and testing of the model is always exclusively by their own model authors, meaning that it is not independent or verifiable and they want us to take their word for it, which we cannot - as they have an axe to grind against open weight models.

This is marketing wrapped around a biased research paper.

Re: Project Glasswing: Securing critical software for the AI era

#417

Post by adi_kurian »

breakingcups wrote: post #224
adi_kurian wrote: post #152

Earlier quoted context omitted.

Opus 4 & 4.1 are still on Vertex+Bedrock @ $75/1mm out. They were used very heavily and in my subjective opinion are better than 4.5 and 4.6.

Interesting, what makes them better to you?

Opus 4, with enough context, could do most all I wanted in a single shot. More often than not, when I had a bad outcome and was frustrated I would realize that I was the problem (in giving improper direction or missing key context).

I also was in a pretty sweet position having a boat load of credits and premo vertex rate limits so I could 'afford' to dump hundreds of thousands of tokens in context all day.

With Opus 4.5 and 4.6, I find I have to steer very actively.

This is comparing using Opus 4 directly rather than comparing the performance of the models in Claude Code for example, or any 'agentic' setup.

Kinda reminds me of 4o vs 4-turbo.

I would imagine they are smaller models.

Re: Project Glasswing: Securing critical software for the AI era

#418

Post by akerl_ »

pilgrim0 wrote: post #388
stale2002 wrote: post #261

Earlier quoted context omitted.

Ok. But mostly its entirely the old software, not the new software, that the bugs are being found in.

Maybe because there’s no critical and widely used software written by LLMs so far? Which says a lot about LLMs are failing to even approach the level of capabilities you would expect from all the hype? The goal has always been, even before LLMs, to find something smarter than our smarter humans. So far the success at that is really minuscule. Humans are still the benchmark, all things considered. Now they’re saying L…

So today, most of the vulnerabilities being found by these tools are in code written by humans. Your hypothesis is that down the road, most of the vulnerabilities will be in code written by LLMs.

What seems more probable is that the same advances that LLMs are shipping to find vulnerabilities will end up baked into developer tooling. So you'll be writing code and using an LLM that knows how to write secure code.

Re: Project Glasswing: Securing critical software for the AI era

#419

Post by woeirua »

wyre wrote: post #382
woeirua wrote: post #315

Earlier quoted context omitted.

I’m pretty close to the point of saying that human intelligence is not special.

I would argue the opposite. It’s gotten us to a point were we can recreate human intelligence from electricity and a bunch of math!

Are you a bot?

Re: Project Glasswing: Securing critical software for the AI era

#420

Post by woeirua »

pilgrim0 wrote: post #400
woeirua wrote: post #76

Earlier quoted context omitted.

You should watch this talk by Nicholas Carlini (security researcher at Anthropic). Everything in the talk was done with Opus 4.6: https://www.youtube.com/watch?v=1sd26pWhfmg

Just a thought: The fact that the found kernel vulnerability went decades without a fix says nothing about the sophistication needed to find it. Just that nobody was looking. So it says nothing about the model’s capability. That LLMs can find vulnerabilities is a given and expected, considering they are trained on code. What worries me is the public buying the idea that it could in any way be a comprehensive security…

Did you even watch the video or read the article?
Post reply on HN