Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability research is cooked

sockpuppet.org

Post reply on HN
Print view View original

161–170 of 181 posts

Re: Vulnerability research is cooked

#161

Post by wredcoll »

intended wrote: post #107
charcircuit wrote: post #74

Earlier quoted context omitted.

You can have an AI agent refactor and improve code quality.

It’s agents all the way down - until you have liability. At some point, it’s going to be someone’s neck on the line, and saying “the agents know” isn’t going to satisfy customers (or in a worst case, courts).

> until you have liability

And are you thinking this going to start happening at some point or what?

The letters I get every other month telling me I now have free credit monitoring because of a personal info breach seems to suggest otherwise.

Re: Vulnerability research is cooked

#162

Post by g947o »

stavros wrote: post #144
hrmtst93837 wrote: post #141

Earlier quoted context omitted.

[flagged]

Any patch you ship can be vetted by the same models, so you can be sure the same models can't find a vulnerability in the attacker's hands. Then it's just a matter of fixing the old vulnerabilities.

"so you can be sure"

Nothing is for sure with LLMs.

Re: Vulnerability research is cooked

#163

Post by stavros »

g947o wrote: post #162
stavros wrote: post #144

Earlier quoted context omitted.

Any patch you ship can be vetted by the same models, so you can be sure the same models can't find a vulnerability in the attacker's hands. Then it's just a matter of fixing the old vulnerabilities.

"so you can be sure" Nothing is for sure with LLMs.

Nothing is for sure with anything.

Re: Vulnerability research is cooked

#164

Post by nickpsecurity »

saagarjha wrote: post #124
nickpsecurity wrote: post #103

Earlier quoted context omitted.

You've never seen the full power of static analysis, dynamic analysis, and test generation. The best examples were always silo'd, academic codebases. If they were combined, and matured, the results would be amazing. I wanted to do that back when I was in INFOSEC. That doesn't even account for lightweight, formal methods. SPARK Ada, Jahob verification system with its many solvers, Design ny Contract, LLM's spitting th…

Silo’d, academic codebases are not under the kind of attacks that commodity software is

The silo'd codebases I was referring to are verification tools they produce. They're used to prevent attacks. Each tool has one or more capabilities others lack. If combined, they'd catch many problems.

Examples: KLEE test generator; combinatorial or path-bases testing; CPAChecker; race detectors for concurrency; SIF information flow control; symbolic execution; Why3 verifier which commercial tools already build on.

Re: Vulnerability research is cooked

#165

Post by tptacek »

cushychicken wrote: post #157
streetfighter64 wrote: post #131

Earlier quoted context omitted.

> Not a POC. An exploit. What's the distinction? A proof of concept is just something that demonstrates that a bug is possible to exploit, by doing so.

Repeatability and/or an actual negative effect. POC generally means “you can demonstrate unintentional behavior”. “Exploit” means you can gain access or do something malicious. It’s a fine line. Author’s point is that the LLM was able to demonstrate some malfeasance, not just unintended consequence. That’s a big deal considering that actual malicious intent generally requires more knowhow than raw POC.

Specifically: the exploit extracted the admin's credentials from the database. A blind SQLI POC would simply demonstrate the existence of a timing channel based on a pathological input.

Re: Vulnerability research is cooked

#166

Post by nostrademons »

lll-o-lll wrote: post #58
batshit_beaver wrote: post #52

Earlier quoted context omitted.

You literally cannot, since ANY changes to code tend to introduce unintended (or at least not explicitly requested) new behaviors.

Eventual convergence? Assuming each defect fix has a 30% chance of introducing a new defect, we keep cycling until done?

The chance of a defect fix introducing a new defect tends to grow linearly with the size of the codebase, since defects are usually caused by the interaction between code and there's now more code to interact with.

If you plot this out, you'll notice that it eventually reaches > 100% and the total number of defects will eventually grow exponentially, as each bugfix eventually introduces more bugs than it fixes. Which is what I've actually observed in 25 years in the software industry. The speed at which new bugs are introduced faster than bugfixes varies by organization and the skill of your software architects - good engineers know how to keep coupling down and limit the space of existing code that a new fix could possibly break. I've seen some startups where they reach this asymptote before bringing the product to market though (needless to say, they failed), and it's pretty common for computer games to become steaming piles of shit close to launch, and I've even seen some Google systems killed and rewritten because it became impossible to make forward progress on them. I call this technical bankruptcy, the end result of technical debt.

Re: Vulnerability research is cooked

#167

Post by sfink »

nickpsecurity wrote: post #104
thadt wrote: post #44

So the intersting question: are we long term safer with "simpler" closer to hardware memory unsafe(ish) environments like Zig, or is the memory safe but more abstract feature set of languages like Rust still the winning direction? If a hypothetical build step is "look over this program and carfully examine the bounds of safety using your deep knowledge of the OS, hardware, language and all the tools that come along w…

They're great at Python and Javascript which have lots of tooling. My idea was to make X-to-safe-lang translators, X initially being Python and Javascript. Let the tools keep generating what they're good at. The simpler translators make it safe and fast. If translated to C or Java, we can use decades worth of tools for static analysis and test generation. While in Python and Javascript, it's easier to analyze and liv…

> My idea was to make X-to-safe-lang translators, X initially being Python and Javascript.

Both of those languages are already safe. Then you talk about translating to C, so you're actually doing a safe-to-unsafe translation. I'm not sure what properties you're checking with the static analysis at that point. I think what would be more important is that your translator maintains safety.

Re: Vulnerability research is cooked

#168

Post by tomjakubowski »

tptacek wrote: post #18
tomjakubowski wrote: post #13

Earlier quoted context omitted.

It sounds like what makes the pipeline in the article effective is the second stage, which takes in the vulnerability reports produced by the first level and confirms or rejects them. The article doesn't say what the rejection rate is there. I don't think the spammers would think to write the second layer, they would most likely pipe the first layer (a more naive version of it too, probably) directly to the issue fee…

There are at least three differences: * Carlini's team used new frontier models that have gotten materially better at finding vulnerabilities (talk to vulnerability researchers outside the frontier labs, they'll echo that). Stenberg was getting random slop from people using random models. * Carlini's process is iterated exhaustively over the whole codebase; he's not starting with a repo and just saying "find me an aw…

Thanks, I hadn't considered the second point.

I guess the broader point I wanted to make is about the people responsible for the deluge of LLM-reported bugs and security vulnerabilities on countless open-source projects (not only on curl): they weren't considerate or thoughtful security researchers, they were spammers looking to raise their profile with fully automated, hands-off open source "contributions". I would expect that the spammers would continue to use whatever lowest common denominator tooling is available, and continue to cause these headaches for maintainers.

That doesn't mean frontier models and tooling built around them aren't genuinely useful to people doing serious security research: that does seem to be the case, and I'm glad for it.

Re: Vulnerability research is cooked

#169

Post by intended »

wredcoll wrote: post #161
intended wrote: post #107

Earlier quoted context omitted.

It’s agents all the way down - until you have liability. At some point, it’s going to be someone’s neck on the line, and saying “the agents know” isn’t going to satisfy customers (or in a worst case, courts).

> until you have liability And are you thinking this going to start happening at some point or what? The letters I get every other month telling me I now have free credit monitoring because of a personal info breach seems to suggest otherwise.

A firm has very different amounts of time, ability and money to spend on following up on broken contracts.

Re: Vulnerability research is cooked

#170

Post by pron »

strogonoff wrote: post #154
pron wrote: post #153

Earlier quoted context omitted.

> exploiting software is someone’s full-time job, whereas the engineers already have one—building it. But the attackers needs to spread their attack over many products, while the engineers only need to defend one. > The newer factor is attackers working for nation-states, being protected by them, and potentially having figurative guns to their heads or at least livelihoods depending on the amount of damage they can d…

> > exploiting software is someone’s full-time job, whereas the engineers already have one—building it. > But the attackers needs to spread their attack over many products, while the engineers only need to defend one. Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely. Realistically, you have people whose job or passion is to develop software, who often work not on one but…

> Are you assuming every piece of software has a dedicated defender team? Strikes me as unlikely.

No, I'm assuming it has maintainers (they play the role of defenders).

> engineers who work on software are simply not that great and dedicated about finding vulnerabilities in it.

Yes, but LLMs help them more than they help the attackers, because the attackers are already security experts. In other words, the LLMs reduce the skill gap rather than increase it. Becoming good at using AI is much easier than becoming good at security.

Post reply on HN