Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small models also found the vulnerabilities that Mythos found

aisle.com

Post reply on HN
Print view View original

81–90 of 372 posts

Re: Small models also found the vulnerabilities that Mythos found

#81

Post by rychu »

dist-epoch wrote: post #11

Anthropic claim is not necessarily that Mythos found vulnerabilities that other models couldn't but that it could easily exploit them while previous models failed to do that: > “Opus 4.6 is currently far better at identifying and fixing vulnerabilities than at exploiting them.” Our internal evaluations showed that Opus 4.6 generally had a near-0% success rate at autonomous exploit development. But Mythos Preview is i…

If that was normal Opus, then it sounds to me like Mythos could be a big model, instruction tuned, but without all the safety/refusal part of training.

Re: Small models also found the vulnerabilities that Mythos found

#82

Post by echelon »

celeritascelery wrote: post #66
johnfn wrote: post #15

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any…

That was my thought exactly. If small models can find these same vulnerabilities, and your company is trying to find vulnerabilities, why didn’t you find them?

Who is spending millions of dollars on small models to find vulns? Nobody else is selling here or has the budget to sell quite like this.

Anthropic spends millions - maybe significantly more.

Then when they know where they are, they spend $20k to show how effective it is in a patch of land.

They engineered this "discovery".

What the small teams are doing is fair - it's just a scaled down version of what Anthropic already did.

Re: Small models also found the vulnerabilities that Mythos found

#83

Post by kilpikaarna »

johnfn wrote: post #15

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any…

Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated.

Have Anthropic actually said anything about the amount of false positives Mythos turned up?

FWIW, I saw some talk on Xitter (so grain of salt) about people replicating their result with other (public) SotA models, but each turned up only a subset of the ones Mythos found. I'd say that sounds plausible from the perspective of Mythos being an incremental (though an unusually large increment perhaps) improvement over previous models, but one that also brings with it a correspondingly significant increase in complexity.

So the angle they choose to use for presenting it and the subsequent buzz is at least part hype -- saying "it's too powerful to release publicly" sounds a lot cooler than "it costs $20000 to run over your codebase, so we're going to offer this directly to enterprise customers (and a few token open source projects for marketing)". Keep in mind that the examples in Nicholas Carlini's presentation were using Opus, so security is clearly something they've been working on for a while (as they should, because it's a huge risk). They didn't just suddenly find themselves having accidentally created a super hacker.

Re: Small models also found the vulnerabilities that Mythos found

#84

Post by replygirl »

rakejake wrote: post #74
celeritascelery wrote: post #66

Earlier quoted context omitted.

That was my thought exactly. If small models can find these same vulnerabilities, and your company is trying to find vulnerabilities, why didn’t you find them?

Maybe they did use small models but you couldn't make the front page of HN with something like this until Anthropic made a big fuss out of it. Or perhaps it is just a question of compute. Not everyone has 20k$ or the GPU arsenal to task models to find vulnerabilities which may/may not be correct? Unless Anthropic makes it known exactly what model + harness/scaffolding + prompt + other engineering they did, these comp…

papers are always coming out saying smaller models can do these amazing and terrifying things if you give them highly constrained problems and tailored instructions to bias them toward a known solution. most of these don't make the front page because people are rightfully unimpressed

Re: Small models also found the vulnerabilities that Mythos found

#85

Post by slopinthebag »

MaxLeiter wrote: post #6

I think they key thing here is they "isolated the relevant code" If the exploits exist in e.g. one file, great. But many complex zerodays and exploits are chains of various bugs/behaviors in complex systems. Important research but I don’t think it dispels anything about Mythos

Did Mythos identify vulnerabilities across files? Afaik Mythos worked the same way, analysing a single file at a time.

Re: Small models also found the vulnerabilities that Mythos found

#86

Post by ang_cire »

SCHiM wrote: post #48
TacticalCoder wrote: post #38

Earlier quoted context omitted.

> That's why their point is what the subheadline says, that the moat is the system, not the model. Can you expand a bit more on this? What is the system then in this case? And how was that model created? By AI? By humans?

You can imagine a pipeline that looks at individual source files or functions. And first "extracts" what is going on. You ask the model: - "Is the code doing arithmetic in this file/function?" - "Is the code allocating and freeing memory in this file/function?" - "Is the code the code doing X/Y/Z? etc etc" For each question, you design the follow-up vulnerability searchers. For a function you see doing arithmetic, yo…

That's funny, this is how I've been doing security testing in my code for a while now, minus the 'taint analysis'. Who knew I was ahead of the game. :P

In all seriousness though, it scares me that a lot of security-focused people seemingly haven't learned how LLMs work best for this stuff already.

You should always be breaking your code down into testable chunks, with sets of directions about how to chunk them and what to do with those chunks. Anyone just vaguely gesturing at their entire repo going, "find the security vulns" is not a serious dev/tester; we wouldn't accept that approach in manual secure coding processes/ SSDLCs.

Re: Small models also found the vulnerabilities that Mythos found

#87

Post by johnfn »

kilpikaarna wrote: post #83
johnfn wrote: post #15

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any…

Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. Have Anthropic actually said anything about the amount of false positives Mythos turned up? FWIW, I saw some talk on Xitter (so grain of salt) about people replicating their result with…

> Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated.

But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none. Both are worthless without human intervention.

I definitely breathed a sigh of relief when I read it was $20,000 to find these vulnerabilities with Mythos. But I also don't think it's hype. $20,000 is, optimistically, a tenth the price of a security researcher, and that shift does change the calculus of how we should think about security vulnerabilities.

Re: Small models also found the vulnerabilities that Mythos found

#88

Post by vmg12 »

mirsadm wrote: post #80
vmg12 wrote: post #50

The technique Anthropic uses was demonstrated by Nicholas Carlini in a talk he gave 2 weeks ago and it's very simple, when asking LLMs to review code, ask them to focus its review on one file in a single session. Here is the video with the timestamp (watch through to ~5:30, they show two different ways of prompting claude). https://youtu.be/1sd26pWhfmg?t=204 https://youtu.be/1sd26pWhfmg?t=273 IMO the big "innovation"…

How is that going to find anything that interacts across files?

My phrasing wasn't clear but you aren't telling it to only look at one specific file but to focus its review on one file. Updated my original comment.

Re: Small models also found the vulnerabilities that Mythos found

#89

Post by tehryanx »

elicash wrote: post #12
epistasis wrote: post #5

> We took the specific vulnerabilities Anthropic showcases in their announcement, isolated the relevant code, and ran them through small, cheap, open-weights models. Those models recovered much of the same analysis. Eight out of eight models detected Mythos's flagship FreeBSD exploit, including one with only 3.6 billion active parameters costing $0.11 per million tokens. Impressive, and very valuable work, but isolat…

This is from the first of the caveats that they list: > Scoped context: Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior"). A real autonomous discovery pipeline starts from a full codebase with no hints. The models' performance here is an upper bound on what they'd achieve in a fully autonomous scan. That said, a well-designed scaffold naturally…

I get what you're saying, but I think this is still missing something pretty critical.

The smaller models can recognize the bug when they're looking right at it, that seems to be verified. And with AISLE's approach you can iteratively feed the models one segment at a time cheaply. But if a bug spans multiple segments, the small model doesn't have the breadth of context to understand those segments in composite.

The advantage of the larger model is that it can retain more context and potentially find bugs that require more code context than one segment at a time.

That said, the bugs showcased in the mythos paper all seemed to be shallow bugs that start and end in a single input segment, which is why AISLE was able to find them. But having more context in the window theoretically puts less shallow bugs within range for the model.

I think the point they are making, that the model doesn't matter as much as the harness, stands for shallow bugs but not for vulnerability discovery in general.

Re: Small models also found the vulnerabilities that Mythos found

#90

Post by appcustodian2 »

mirsadm wrote: post #80
vmg12 wrote: post #50

The technique Anthropic uses was demonstrated by Nicholas Carlini in a talk he gave 2 weeks ago and it's very simple, when asking LLMs to review code, ask them to focus its review on one file in a single session. Here is the video with the timestamp (watch through to ~5:30, they show two different ways of prompting claude). https://youtu.be/1sd26pWhfmg?t=204 https://youtu.be/1sd26pWhfmg?t=273 IMO the big "innovation"…

How is that going to find anything that interacts across files?

I would think that it is still capable of exploring the codebase and reading other related files like any other coding agent already does.
Post reply on HN