Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI is breaking two vulnerability cultures

jefftk.com

Post reply on HN
Print view View original

81–90 of 188 posts

Re: AI is breaking two vulnerability cultures

#81

Post by jfk13 »

0xbadcafebee wrote: post #67

We need automated patch and release cycles. So far we've relied on incredibly slow manual processes to accept reports, investigate, verify, patch, and prepare releases. Releasing a fix often takes months. This is way too slow when attackers can just churn out new exploits in hours. We need to iterate on value chain bottlenecks to lower Mean Time To Patch . We should be able to turn around a bug report to a patched pr…

Sounds like you're expecting the AI-based tools that are finding bugs to also provide fixes.

I've been dealing with a bunch of AI-generated (or at least -assisted) vulnerability reports lately. In many cases the reports include proposed patches to fix the issues.

It's been..... interesting. In many cases, the analysis provided in the report has been accurate and helpful. In some cases, the proposed patches have also been good, and we've accepted them with minimal or no changes.

In other cases, despite finding a valid issue, and even providing a good analysis of the problem, the AI tool's suggested patch has been, quite simply, wrong.

Careful review from somebody who really _understands_ the code -- and the wider context in which it is operating -- is still absolutely necessary. That's not always going to happen in an hour.

Re: AI is breaking two vulnerability cultures

#83

Post by jefftk »

awesome_dude wrote: post #24
ragall wrote: post #21

Earlier quoted context omitted.

> How do you know? We know because we could see the effects of the average rate of vulnerabilities discovery and exploitation, and it's definitely going up very fast. Until recently, vulnerabilities were relatively hard to find, and finding them was done by a very restricted group of people world-wide, which made them quite valuable. Not any more.

That's correlation, not causation. It could equally be argued that the AI slop that's being produced makes for a lot more vulnerabilities being shipped. The bigger target makes for the easier discovery.

It's likely varies enormously between projects. Linux remains extremely low in slop, and the vulnerabilities being fixed are quite old, so it's improving. Many vibe coded projects are very sloppy, and are adding a lot of vulnerabilities.

Total number of vulnerabilities likely goes up over time weighting all projects equally, but goes down over time weighting by usage.

Re: AI is breaking two vulnerability cultures

#84

Post by jefftk »

Worf wrote: post #74
jefftk wrote: post #70

Earlier quoted context omitted.

> 90 days is ridiculous, especially for companies It depends on the kind of vulnerability, but sometimes in order to fix a problem, you need to do an enormous amount of software engineering. Which needs to be done to a very high standard, because the expectation is that people will push security patches more or less immediately to production. Of course, this only works if no one else is likely to discover the vulnera…

The company can almost always shut down their service until they fix it. They'll lose money and their customers could also lose money if they depend on the service. That's the price they'll have to pay. Otherwise, they should either work frantically 24/7 to fix the vuln or if they can't, they should accept the fact that they've pushed code without any regard for security and bear the consequences. Why do we need to p…

How would you apply this logic to something like https://meltdownattack.com ? The vulnerability was in hardware, discovered by companies that make user level software, and mitigated by changes to OS kernels.

Re: AI is breaking two vulnerability cultures

#85

Post by dakolli »

woah wrote: post #33
miki123211 wrote: post #28

AI will shorten update windows dramatically. 2026 is the worst year to be thinking about dependency cooldowns, we need to think about dependency warmups instead. Soon, there will be no such thing as a safe way to disclose a vulnerability in an open source project. Centralized SaaS will have a major security advantage here.

You could have a web of trust where Linux-using organizations each spend $x continuously scanning and patching their own dependencies with AI, and sending each other patches and scans.

LLMs aren't capable of doing this, and never will be no matter what Anthropic tries tell you.

Re: AI is breaking two vulnerability cultures

#86

Post by freeqaz »

This is exactly what happened with Log4Shell.

Day -X + 1: Engineer at Alibaba finds the vuln and tells Apache. Patch is pushed to git while new release is coordinated.

Day -X: A black hat sees commits fixing the bug. Attacks start happening.

Day 0: Memes start circulating in Minecraft communities of people crashing servers. Some logs are shared on Twitter, especially in China, of people getting pwned.

Day 0 + ~4 hours: My friend DMs me a meme on Twitter. I look up to find the CVE. Doesn't exist. My friend and I reproduce the exploit and write up a blog post about it. (We name it Log4Shell to differentiate it from a different, older log4j RCE vuln)

Day ~1: Media starts picking it up. Apache is forced to release patches faster in response. CVE is actually published to properly allow security scanners to identify it.

Today: AI makes this happen faster and more consistently. Patches probably should be kept private until a coordinated disclosure happens post-testing and CVE being published?

Hard to say what the right move is, but this is gonna be happening a lot over the next 1-3 years. Lots of companies are going to be getting cooked until AI helps us patch faster than attackers can exploit these fresh 0-days.

Re: AI is breaking two vulnerability cultures

#87

Post by awesome_dude »

jefftk wrote: post #83
awesome_dude wrote: post #24

Earlier quoted context omitted.

That's correlation, not causation. It could equally be argued that the AI slop that's being produced makes for a lot more vulnerabilities being shipped. The bigger target makes for the easier discovery.

It's likely varies enormously between projects. Linux remains extremely low in slop, and the vulnerabilities being fixed are quite old, so it's improving. Many vibe coded projects are very sloppy, and are adding a lot of vulnerabilities. Total number of vulnerabilities likely goes up over time weighting all projects equally, but goes down over time weighting by usage.

I mean - you're spot on - which is why I'd be more inclined to ask for actual metrics rather than feels/vibes, and I'd be very clear that the information I was basing my thinking on has enormous pitfalls.

This is the basis for "correlation points to possibly fertile grounds for an investigation"

Re: AI is breaking two vulnerability cultures

#88

Post by fragmede »

dakolli wrote: post #85
woah wrote: post #33

Earlier quoted context omitted.

You could have a web of trust where Linux-using organizations each spend $x continuously scanning and patching their own dependencies with AI, and sending each other patches and scans.

LLMs aren't capable of doing this, and never will be no matter what Anthropic tries tell you.

Mozilla seems to think it can.

https://blog.mozilla.org/en/privacy-security/ai-security-zer...

Re: AI is breaking two vulnerability cultures

#89

Post by fragmede »

oytis wrote: post #36

The old saying of Tony Hoare about no obvious bugs vs obviously no bugs holds in the age of LLMs more than ever

> There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies." — Tony Hoare

for those (like me) who hadn't seen it before.

Re: AI is breaking two vulnerability cultures

#90

Post by globalnode »

ReptileMan wrote: post #79
grog454 wrote: post #44

Earlier quoted context omitted.

> It has been over a decade since any ordinary off-the-shelf closed-source software was meaningfully obscured from serious adversaries. Probably goes without saying but the last line of defense is not deploying your software publicly and instead relying on server-client architectures to do anything. Maybe this will be more common as vulnerabilities are more easily detected and exploited. Of course its not always feas…

>Only the undeployed server code has remained private. How easy to do you this is for LLM to build decent emulator of the server in question by just observing what you send and what you get as response?

not sure why downvoted. server emulators will become faster to make. protocol analysis will become faster as well.
Post reply on HN