Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small models also found the vulnerabilities that Mythos found

aisle.com

Post reply on HN
Print view View original

161–170 of 372 posts

Re: Small models also found the vulnerabilities that Mythos found

#161

Post by hakanderyal »

chirau wrote: post #13

Their isolation approach is totally different from Mythos approach though. Mythos had to evaluate whole code bases rather than isolated sections. It's like saying one dog walked into the Amazon jungle and found a tennis ball and then another team isolated a 1 square kilometer radius that they knew the ball was definitely in and found the same ball.

Even that would be more meaningful test. They basically coated the ball with a strong smell, then they prepped the dog with that smell, then set it loose in a 5x5 meter area.

"Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior")."

Re: Small models also found the vulnerabilities that Mythos found

#163

Post by johnfn »

glerk wrote: post #146
johnfn wrote: post #15

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any…

I'm having trouble finding this info (I assume they won't publish it), but could the secret sauce be much larger and more readily accessible context window? OpenBSD's code is in the 10s of millions of lines. Being able to hold all of it in context would make bug finding much easier.

You can look at some of the bugs, if you'd like. They are (at least the ones I looked at) fairly self-contained, scoped to a single function, a hundred lines or less. There's no need for a massive amount of context.

Re: Small models also found the vulnerabilities that Mythos found

#164

Post by sumeno »

mnicky wrote: post #155
johnfn wrote: post #87

Earlier quoted context omitted.

> Wasn't the scaffolding for the Mythos run basically a line of bash that loops through every file of the codebase and prompts the model to find vulnerabilities in it? That sounds pretty close to "any gold there?" to me, only automated. But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none.…

Also, what is $20,000 today can be $2000 next year. Or $20... See e.g. https://epoch.ai/data-insights/llm-inference-price-trends/

Or $200,000 for consumers when they have to make a profit

Re: Small models also found the vulnerabilities that Mythos found

#165

Post by argee »

johnfn wrote: post #154
sweezyjeezy wrote: post #125

Earlier quoted context omitted.

> But the entire value is that it can be automated. If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns. Or none. 'Or none' is ruled out since it found the same vulnerability - I agree that there is a question on precision on the smaller model, but barring further analysis it just feels like '9500' is pure vibes from yourself? Also (out of intere…

Admittedly just vibes from me, having pointed small models at code and asked them questions, no extensive evaluation process or anything. For instance, I recall models thinking that every single use of `eval` in javascript is a security vulnerability, even something obviously benign like `eval("1 + 1")`. But then I'm only posting comments on HN, I'm not the one writing an authoritative thinkpiece saying Mythos actual…

With LLMs (and colleagues) it might be a legitimate problem since they would load that eval into context and maybe decide it’s an acceptable paradigm in your codebase.

Re: Small models also found the vulnerabilities that Mythos found

#166

Post by pardon_me »

slibhb wrote: post #124

The best way to think of Anthropic's communication about Mythos is as advertisement. It's basically "our model is too smart to release" which suggests they're ahead of OpenAI (without proof)

The whole company is like that. If things were as amazing as advertised, they wouldn't even need to advertise. Or to release models to the public at all.

Re: Small models also found the vulnerabilities that Mythos found

#167

Post by amazingamazing »

johnfn wrote: post #130
amazingamazing wrote: post #91

Earlier quoted context omitted.

Citation needed for basically all of this. You basically are creating a double standard for small models vs mythos…

The citation is the Anthropic writeup.

They did not say what you are saying…

> If you try to automate a small model to look for vulnerabilities over 10,000 files, it's going to say there are 9,500 vulns.

Re: Small models also found the vulnerabilities that Mythos found

#168

Post by lmeyerov »

johnfn wrote: post #15

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any…

Instead of scanning more code, afaict what you seem to want is instead, scan on the same small area, and compare on how many FPs are found there. A common measure here is what % of the reported issues got labeled as security issues and fixed. I don't see Mythos publishing on relative FP rate, so dunno how to compare those. Maybe something substantively changed?

At the same time, I'm not sure that really changes anything because I don't see a reason to believe attacks are constrained by the quality of source code vulnerability finding tools, at least for the last 10-15 years after open source fuzzing tools got a lot better, popular, and industrialized.

This might sound like a grumpy reply, but as someone on both sides here, it's easy to maintain two positions:

1. This stuff is great, and doing code reviews has been one of my favorite claude code use cases for a year now, including security review. It is both easier to use than traditional tools, and opens up higher-level analysis too.

2. Finding bugs in source code was sufficiently cheap already for attackers. They don't need the ease of use or high-level thing in practice, there's enough tooling out there that makes enough of these. Likewise, groups have already industrialized.

There's an element of vuln-pocalypse that may be coming with the ease of use going further than already happening with existing out-of-the-box blackbox & source code scanning tools . That's not really what I worry about though.

Scarier to me, instead, is what this does to today's reliance on human response. AI rapidly industrializes what how attackers escalate access and wedge in once they're in. Even without AI, that's been getting faster and more comprehensive, and with AI, the higher-level orchestration can get much more aggressive for much less capable people. So the steady stream of existing vulns & takeovers into much more industrialized escalations is what worries me more. As coordination keeps moving into machine speed, the current reliance on human response is becoming less and less of an option.

Re: Small models also found the vulnerabilities that Mythos found

#169

Post by rakel_rakel »

johnfn wrote: post #15

The Anthropic writeup addresses this explicitly: > This was the most critical vulnerability we discovered in OpenBSD with Mythos Preview after a thousand runs through our scaffold. Across a thousand runs through our scaffold, the total cost was under $20,000 and found several dozen more findings. While the specific run that found the bug above cost under $50, that number only makes sense with full hindsight. Like any…

Spending $20000 (and whatever other resources this thing consumes) on a denial of service vulnerability in OpenBSD seems very off balance to me.

Given the tone with which the project communicates discussing other operating systems approaches to security, I understand that it can be seen as some kind of trophy for Mythos. But really, searching the number of erratas on the releases page that include "could crash the kernel" makes me think that investing in the OpenBSD project by donating to the foundation would be better than using your closed source model for peacocking around people who might think it's harder than it is to find such a bug.

Re: Small models also found the vulnerabilities that Mythos found

#170

Post by decidu0us9034 »

SCHiM wrote: post #48
TacticalCoder wrote: post #38

Earlier quoted context omitted.

> That's why their point is what the subheadline says, that the moat is the system, not the model. Can you expand a bit more on this? What is the system then in this case? And how was that model created? By AI? By humans?

You can imagine a pipeline that looks at individual source files or functions. And first "extracts" what is going on. You ask the model: - "Is the code doing arithmetic in this file/function?" - "Is the code allocating and freeing memory in this file/function?" - "Is the code the code doing X/Y/Z? etc etc" For each question, you design the follow-up vulnerability searchers. For a function you see doing arithmetic, yo…

I think there is already papers and presentations on integrating these kind of iterative code understanding/verificaiton loops in harnesses. There may be some advantages over fuzzing alone. But I think the cost-benefit analysis is a lot more mixed/complex than anthropic would like people to believe. Sure you need human engineers but it's not like insurmountably hard for a non-expert to figure out
Post reply on HN