Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small models also found the vulnerabilities that Mythos found

aisle.com

Post reply on HN
Print view View original

261–270 of 372 posts

Re: Small models also found the vulnerabilities that Mythos found

#261

Post by felipeerias »

StrauXX wrote: post #222

A lot of comments here are dismissing this post because the relevant code was isolated. But thats the exact same thing Anthropic did with Mythos! They describe their (very lean) harness in the Anthropic Red Mythos blog post. The harness first assigns each file in the given codebase an importance value. Then points claude code at the cpdebase with a prompt stating that it should focus on that file. It spawns a claude…

From the article:

> Our tests gave models the vulnerable function directly, often with contextual hints (e.g., "consider wraparound behavior").

Re: Small models also found the vulnerabilities that Mythos found

#262

Post by eiens »

johnfn wrote: post #248
jorvi wrote: post #227

Earlier quoted context omitted.

My proof-in-pudding test is still the fact that we haven't seen gigantic mass firings at tech companies, nor a massive acceleration on quality or breadth (not quantity!) of development. Microsoft has been going heavy on AI for 1y+ now. But then they replace their cruddy native Windows Copilot application with an Electron one. If tests and dev only has marginal cost now, why aren't they going all in on writing extreme…

Your proof-in-pudding test seems to assume that AI is binary -- either it accelerates everyone's development 100x ("let's rewrite every app into bug-free native applications") or nothing ("there hasn't been anything to show for that in years"). I posit reality is somewhere in between the two.

LLM’s are capable of searching information spaces and generating some outputs that one can use to do their job.

But it’s not taking anyone’s job, ever. People are not bots, a lot of the work they do is tacit and goes well beyond the capabilities and abilities of llm’s.

Many tech firms are essentially mature and are currently using too much labour. This will lead to a natural cycle of lay offs if they cannot figure out projects to allocate the surplus labour. This is normal and healthy - only a deluded economist believes in ‘perfect’ stuff.

Re: Small models also found the vulnerabilities that Mythos found

#263

Post by morpheuskafka »

Everyone is commenting that this doesn't count because they pointed it at the specific files that Mythos already found vulnerable.

But sometimes you do know where vulnerabilities are and still don't know what they are. For example, an update may be released in beta changing the part of the Mac or Windows kernel or some app, but they haven't published the CVE yet. If locally runnable (even with significant compute costs) LLMs can find and exploit it based on either the location of the changed file or the actual diff of the compiled output, we could see exploits before the update ever went to production?

Re: Small models also found the vulnerabilities that Mythos found

#264

Post by tokioyoyo »

muyuu wrote: post #255

I think the "Mythos" name is genius. The people at Anthropic make a bunch of claims and the public is expected to just believe them without any possibility of testing those claims or reproducing those results, and since so many people are invested in this saviour for the Global economy, or in the industry in general, or in hype to feed their engagement-based income sources, then there is faith to spare. Meanwhile thi…

There are two possibilities:

a) Anthropic is lying, and every company that is collaborating on vulnerability squishing project is an accomplice in this big lie b) Anthropic has then goldest gold of the shovels to sell to people, which is actually useful for enterprises

Everyone, including Ant, understands that other companies will catch up in terms of model strength. So it’s a damned if you do, damned if you don’t position wrt releasing it to the public.

Re: Small models also found the vulnerabilities that Mythos found

#265

Post by jurschreuder »

All these models will completely mess up your code if you let them.

And if they constantly scan your code with various settings and updates you will spend hours a day reading, trying to understand locally coherent but structurally incoherent vibes trying to pinpoint the exact reasoning flaw. Exhausting.

Re: Small models also found the vulnerabilities that Mythos found

#266

Post by eiens »

MidnightRider39 wrote: post #236
jorvi wrote: post #227

Earlier quoted context omitted.

My proof-in-pudding test is still the fact that we haven't seen gigantic mass firings at tech companies, nor a massive acceleration on quality or breadth (not quantity!) of development. Microsoft has been going heavy on AI for 1y+ now. But then they replace their cruddy native Windows Copilot application with an Electron one. If tests and dev only has marginal cost now, why aren't they going all in on writing extreme…

Leadership is also a very human thing. I think most people would balk at the idea of being led by an LLM. One of the main functions of leaders (should be) is to assume responsibility for decisions and outcomes. A computer cant do that. And finally why should someone in power choose to replace themselves?

Someone in power doesn’t get to choose - the board of directors do. Who’s job is to act in the best interest of shareholders.

Firms tend to follow peers in an industry - once one blinks the rest follow.

Re: Small models also found the vulnerabilities that Mythos found

#267

Post by apgwoz »

usef- wrote: post #250
apgwoz wrote: post #244

Earlier quoted context omitted.

Why? They claim this small model found a bug given some context. I assume the context wasn’t “hey! There’s a very specific type of bug sitting in this function when certain conditions are met.” We keep assuming that the models need to get bigger and better, and the reality is we’ve not exhausted the ways in which to use the smaller models. It’s like the Playstation 2 games that came out 10 years later. Well now all t…

If this were true, we're essentially saying that no one tried to scan vulnerabilities using existing models, despite vulnerabilities being extremely lucrative and a large professional industry. Vulnerability research has been one of the single most talked about risks of powerful AI so it wasn't exactly a novel concept, either. If it is true that existing models can do this, it would imply that LLMs are being under ma…

I use the models to look for vulnerabilities all the time. I find stuff often. Have I tried to do build a new harness, or develop more sophisticated techniques? No. I suspect there are some spending lots of tokens developing more sophisticated strategies, in the same way software engineers are seeking magical one-shot harnesses.

Re: Small models also found the vulnerabilities that Mythos found

#268

Post by MidnightRider39 »

eiens wrote: post #266
MidnightRider39 wrote: post #236

Earlier quoted context omitted.

Leadership is also a very human thing. I think most people would balk at the idea of being led by an LLM. One of the main functions of leaders (should be) is to assume responsibility for decisions and outcomes. A computer cant do that. And finally why should someone in power choose to replace themselves?

Someone in power doesn’t get to choose - the board of directors do. Who’s job is to act in the best interest of shareholders. Firms tend to follow peers in an industry - once one blinks the rest follow.

The board of directors are also people in power - why not replace them with an LLM as well if it works so well for CEOs?

Re: Small models also found the vulnerabilities that Mythos found

#269

Post by danieltanfh95 »

tptacek wrote: post #257
danieltanfh95 wrote: post #253

Earlier quoted context omitted.

People really lack imagination. The point here is that a dedicated attacker with a good harness and really cheap models can run the attack regardless. It's like portscan/url search attacks. They could run all of these against all codebases and clients. However, on the flip side, this also means we could run cheap models against every PR made, and do a thorough red-team security review. None of these requires mythos.…

That is a point. It might even be true. But showing a small model an example of vulnerable code and asking to confirm that it is vulnerable code isn't evidence for that point!

No, it is evidence for that point. You could just rattle off every possible vulnerability and have the cheap model scan for it in the harness through a loop.

Note that I say cheap, not small, because small models may lack the reasoning needed, but some models are cheap enough but retain enough reasoning (ala Sonnet 3.7+)

Re: Small models also found the vulnerabilities that Mythos found

#270

Post by tptacek »

danieltanfh95 wrote: post #269
tptacek wrote: post #257

Earlier quoted context omitted.

That is a point. It might even be true. But showing a small model an example of vulnerable code and asking to confirm that it is vulnerable code isn't evidence for that point!

No, it is evidence for that point. You could just rattle off every possible vulnerability and have the cheap model scan for it in the harness through a loop. Note that I say cheap, not small, because small models may lack the reasoning needed, but some models are cheap enough but retain enough reasoning (ala Sonnet 3.7+)

That's not what they did.
Post reply on HN