Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kernel code removals driven by LLM-created security reports

lwn.net

Post reply on HN
Print view View original

11–20 of 130 posts

Re: Kernel code removals driven by LLM-created security reports

#11

Post by stratos123 »

ferguess_k wrote: post #7

Are we already in the time, or close to the time, that well-trained LLMs are more efficient in finding security holes than all but the best developers out there, even for OS kernel code? Can someone educate me on this?

In terms of quantity, definitely yes (a single person managing a swarm of Opusi can already find much more real bugs than a security researcher, hence the rise in reports).

In terms of quality ("are there bugs that professional humans can't see at any budget but LLMs can?") - it's not very clear, because Opus is still worse than a human specialist, but Mythos might be comparable. We'll just have to wait and see what results Project Glasswing gets.

Either way, cybersecurity is going to get real weird real soon, because even slightly-dumb models can have a large effect if they are cheap and fast enough.

EDIT: Mozilla thinks "no" to the second question, by the way: "Encouragingly, we also haven’t seen any bugs that couldn’t have been found by an elite human researcher.", when talking about the 271 vulnerabilities recently found by Mythos. https://blog.mozilla.org/en/firefox/ai-security-zero-day-vul...

Re: Kernel code removals driven by LLM-created security reports

#12

Post by cozzyd »

Seems like there should be some "level of maintenance" metric for modules and distros can pick which they include by default and which are packaged separately based on what they care about. Arch users will build the world but an EL user who needs an unmaintained module would have to explicitly install kmod-isdn or even build it themselves

Re: Kernel code removals driven by LLM-created security reports

#14

Post by DanielHB »

stratos123 wrote: post #11
ferguess_k wrote: post #7

Are we already in the time, or close to the time, that well-trained LLMs are more efficient in finding security holes than all but the best developers out there, even for OS kernel code? Can someone educate me on this?

In terms of quantity, definitely yes (a single person managing a swarm of Opusi can already find much more real bugs than a security researcher, hence the rise in reports). In terms of quality ("are there bugs that professional humans can't see at any budget but LLMs can?") - it's not very clear, because Opus is still worse than a human specialist, but Mythos might be comparable. We'll just have to wait and see what…

There is also a huge surface area of security problems that can't happen in practice due to how other parts of the code work. A classic example is unsanitized input being used somewhere where untrusted users can't inject any input.

Being flooded with these kind of reports can make the actual real problems harder to see.

Re: Kernel code removals driven by LLM-created security reports

#15

Post by goalieca »

staticassertion wrote: post #2

They can't maintain the code so they are no longer going to maintain the code.

Maybe attackers would focus on these unused bits for very niche products, but generally no one would waste their time.

In general, drivers make up the largest attack surface in the kernel and many of them are just along for the ride rather than being actively maintained and reviewed by researchers.

Re: Kernel code removals driven by LLM-created security reports

#16

Post by bastawhiz »

sigmoid10 wrote: post #3
staticassertion wrote: post #2

They can't maintain the code so they are no longer going to maintain the code.

Seems like this should have happened anyways and LLMs just finally forced them to admit it.

You're being downvoted but I think you're right in a lot of ways. If you read through the patches for some of the removals, the reasons come down to:

- Nobody is familiar with the code

- Almost all of the recent fixes are from static analysis

- Nobody is even sure if anyone uses the code

This feels a lot like CPython culling stdlib modules and making them pypi packages. The people who rely on those things have a little bit of extra work if they want a recent kernel version, and everyone else benefits (directly or indirectly) by way of there being less stuff that needs attention.

Re: Kernel code removals driven by LLM-created security reports

#17

Post by chuckadams »

stratos123 wrote: post #11
ferguess_k wrote: post #7

Are we already in the time, or close to the time, that well-trained LLMs are more efficient in finding security holes than all but the best developers out there, even for OS kernel code? Can someone educate me on this?

In terms of quantity, definitely yes (a single person managing a swarm of Opusi can already find much more real bugs than a security researcher, hence the rise in reports). In terms of quality ("are there bugs that professional humans can't see at any budget but LLMs can?") - it's not very clear, because Opus is still worse than a human specialist, but Mythos might be comparable. We'll just have to wait and see what…

> Opusi

The plural of "Opus" is "Opera". Might be a tad confusing tho :)

Re: Kernel code removals driven by LLM-created security reports

#18

Post by jcalvinowens »

ferguess_k wrote: post #7

Are we already in the time, or close to the time, that well-trained LLMs are more efficient in finding security holes than all but the best developers out there, even for OS kernel code? Can someone educate me on this?

My experience with these tools is that they generate absolutely enormous amounts of insidiously wrong false positives, and it actually takes a decent amount of skill to work through the 99% which is garbage with any velocity.

Of course some people don't do that, and send all the reports anyway... and then scream from the hilltops about how incredible LLMs are when by sheer luck one happens to be right. Not only is that blatant p-hacking, it's incredibly antisocial.

It's disingenuous marketing speak to say LLMs are "finding" any security holes at all: they find a thousand hypotheticals of which one or two might be real. A broken clock is right twice a day.

Re: Kernel code removals driven by LLM-created security reports

#20

Post by stratos123 »

traceroute66 wrote: post #10
ferguess_k wrote: post #7

Are we already in the time, or close to the time, that well-trained LLMs are more efficient in finding security holes than all but the best developers out there, even for OS kernel code? Can someone educate me on this?

> well-trained LLMs are more efficient in finding security holes than all but the best developers out there, even for OS kernel code? No. Like everything else an LLM touches, it is prone to slop and hallucinations. You still need someone who knows what they are doing to review (and preferably manually validate) the findings. What all this recent hype carefully glosses over is the volume of false-positives. I guarante…

A 0% false-positive rate is not necessary for LLM-powered security review to be a big deal. It was worthless a few months ago, when the models were terrible at actually finding vulnerabilities and so basically all the reports were confabulated, with a false positive rate of >95%. Nowadays things are much better - see e.g. [1] by a kernel maintainer.

Another way to see this is that you mentioned "LLM found this serious bug in Firefox", but the actual number in that Mozilla report [2] was 14 high-severity bugs, and 90 minor ones. However you look at it, it's an impressive result for a security audit, and I dount that the Antropic team had to manually filter out hundreds-to-thousands of false-positives to produce it.

They did have to manually write minimal exploits for each bug, because Opus was bad at it[3]. This is a problem that Mythos doesn't have. With access to Mythos, to repeat the same audit, you'd likely just need to make the model itself write all the exploits, which incidentally would also filter out a lot of the false positives. I think the hype is mostly justified.

[1] https://lwn.net/Articles/1065620/

[2] https://blog.mozilla.org/en/firefox/hardening-firefox-anthro...

[3] https://www.anthropic.com/news/mozilla-firefox-security

Post reply on HN