Small models also found the vulnerabilities that Mythos found • Hacker News Board

Re: Small models also found the vulnerabilities that Mythos found

#151

Post by eiens » Sat, Apr 11, 2026, 8:10 PM UTC

SoftTalker wrote: post #64

rvnx wrote: post #37

[flagged]

The patterns of buggy code are well trained.

The bigger point of focus is that the enterprise value accrues to assets associated with software production.

What happened to all that nonsense about LLM’s solving physics, science etc? Lmao that certainly is not happening.

The natural home of LLM’s is in relation to software production.

The question is can Anthropic and OAI survive? If OAI can’t make their entry into the ad business work then they will fight over the same territory. Meaning both of their chances of survival drop as Google who is a monster in relation to software production will not only seek to kill them but buy their GPU’s at a discounted price.

Re: Small models also found the vulnerabilities that Mythos found

#152

Post by lukev » Sat, Apr 11, 2026, 8:14 PM UTC

johnfn wrote: post #15

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any…

This is a really interesting point though -- it's really scaffold-dependent.

Because for the same price, you could point the small model at each function, one by one, N times each, across N prompts instructing it to look for a specific class of issue.

It's not that there's no difference between models, but it's hard to judge exactly how much difference there is when so much depends on the scaffold used. For a properly scientific test, you'd need to use exactly the same one.

Which isn't possible when Anthropic won't release the model.

Re: Small models also found the vulnerabilities that Mythos found

#153

Post by siva7 » Sat, Apr 11, 2026, 8:14 PM UTC

johnfn wrote: post #87

kilpikaarna wrote: post #83

Earlier quoted context omitted.

Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. Have Anthropic actually said anything about the amount of false positives Mythos turned up? FWIW, I saw some talk on Xitter (so grain of salt) about people replicating their result with…

> Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none.…

Except you would need about 10,000 security researches in parallel to inspect the whole FreeBSD codebase. So about 200 million dollars at least.

Re: Small models also found the vulnerabilities that Mythos found

#154

Post by johnfn » Sat, Apr 11, 2026, 8:18 PM UTC

sweezyjeezy wrote: post #125

johnfn wrote: post #87

Earlier quoted context omitted.

> Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none.…

> But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none. 'Or none' is ruled out since it found the same vulnerability - I agree that there is a question on precision on the smaller model, but barring further analysis it just feels like '9500' is pure vibes from yourself? Also (out of intere…

Admittedly just vibes from me, having pointed small models at code and asked them questions, no extensive evaluation process or anything. For instance, I recall models thinking that every single use of `eval` in javascript is a security vulnerability, even something obviously benign like `eval("1 + 1")`. But then I'm only posting comments on HN, I'm not the one writing an authoritative thinkpiece saying Mythos actually isn't a big deal :-)

Re: Small models also found the vulnerabilities that Mythos found

#155

Post by mnicky » Sat, Apr 11, 2026, 8:20 PM UTC

johnfn wrote: post #87

kilpikaarna wrote: post #83

Earlier quoted context omitted.

Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. Have Anthropic actually said anything about the amount of false positives Mythos turned up? FWIW, I saw some talk on Xitter (so grain of salt) about people replicating their result with…

> Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none.…

Also, what is $20,000 today can be $2000 next year. Or $20...

See e.g. https://epoch.ai/data-insights/llm-inference-price-trends/

Re: Small models also found the vulnerabilities that Mythos found

#156

Post by _pdp_ » Sat, Apr 11, 2026, 8:21 PM UTC

  find ./ \( -name '*.c' -o -name '*.cpp' \) -exec agent.sh -p "can you spot any vulnerabilities in {}" \;

Re: Small models also found the vulnerabilities that Mythos found

#157

Post by Departed7405 » Sat, Apr 11, 2026, 8:21 PM UTC

antirez wrote: post #16

Congrats: completely broken methodology, with a big conflict of interest. Giving specific bug hints, with an isolated function that is suspected to have bugs, is not the same task, NOR (crucially) is a task you can decompose the bigger task into. It is basically impossible to segment code in pieces, provide pieces to smaller models, and expect them to find all the bugs GPT 5.4 or other large models can find. Second:…

Exactly, this is so flawed. Anthropic themselves said they only reported Give open models an environment (prior to Feb 15- so no Mythos-discovered vulns are patche) of Linux and see how many vulnerabilities it can find. Then put it in a sandbox and see if it can escape and send you an e-mail.

Re: Small models also found the vulnerabilities that Mythos found

#158

Post by sfink » Sat, Apr 11, 2026, 8:28 PM UTC

lordofgibbons wrote: post #33

Without showing false-positive rates this analysis is useless. If your model says every line if your code has a bug, it will catch 100% of the bugs, but it's not useful at all. They tested false-positives with only a single bug... I'm not defending anthropic and openai either. Their numbers are garbage too since they don't produce false-positive rates either. Why is this "analysis" making the rounds?

Yes, and in this case they pointed at the function, so a 1-bit model ("yes") would be correct. But it's not that bad. First, they included a test with a false positive. The small models got it right, Opus got it wrong. Second, they asked for an analysis. Look for "Exploitation reasoning, single follow-up prompt:" in the post. It's hard to tell how good they were at a glance, though apparently the full logs are available so you could pull them up.

Anyway, it seems like they erred in the up-front claim "small models found the vulnerability we pointed directly at!", but the findings are at least somewhat stronger if you read through the details.

The small models didn't match Mythos at exploitation. They suggested plausible exploits, but didn't actually try them out so I can't tell if they would have worked. Deepseek R1's sounds pretty convincing to me, but I'm not a good judge. (I'm more in the space of accidentally writing vulnerabilities, not seeking them out or exploiting them. Well, ok, I have a static analysis that finds some, at least.)

Re: Small models also found the vulnerabilities that Mythos found

#159

Post by enraged_camel » Sat, Apr 11, 2026, 8:29 PM UTC

tptacek wrote: post #126

integralid wrote: post #98

Earlier quoted context omitted.

>Or none We already know this is not true, because small models found the same vulnerability.

No, they didn't. They distinguished it, when presented with it. Wildly different problem.

Yeah. And it is totally depressing that this article got voted to the top of the front page. It means people aren’t capable of this most basic reasoning so they jumped on the “aha! so the mythos announcement was just marketing!!”

Re: Small models also found the vulnerabilities that Mythos found

#160

Post by idopmstuff » Sat, Apr 11, 2026, 8:33 PM UTC

sweezyjeezy wrote: post #125

johnfn wrote: post #87

Earlier quoted context omitted.

> Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none.…

> But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none. 'Or none' is ruled out since it found the same vulnerability - I agree that there is a question on precision on the smaller model, but barring further analysis it just feels like '9500' is pure vibes from yourself? Also (out of intere…

> 'Or none' is ruled out since it found the same vulnerability

It's not, though. It wasn't asked to find vulnerabilities over 10,000 files - it was asked to find a vulnerability in the one particular place in which the researchers knew there was a vulnerability. That's not proof that it would have found the vulnerability if it had been given a much larger surface area to search.