Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI is breaking two vulnerability cultures

jefftk.com

Post reply on HN
Print view View original

41–50 of 188 posts

Re: AI is breaking two vulnerability cultures

#41

Post by tptacek »

This has been a very long time coming and the crackup we're starting to see was predicted long before anyone knew what an LLM is.

The catalyst is the shift towards software transparency: both the radically increased adoption of open source and source-available software, and the radically improved capabilities of reversing and decompilation tools. It has been over a decade since any ordinary off-the-shelf closed-source software was meaningfully obscured from serious adversaries.

This has been playing out in slow motion ever since BinDiff: you can't patch software without disclosing vulnerabilities. We've been operating in a state of denial about this, because there was some domain expertise involved in becoming a practitioner for whom patches were transparently vulnerability disclosures. But AIs have vaporized the pretense.

It is now the case that any time something gets merged into mainline Linux, several different organizations are feeding the diffs through LLM prompts aggressively evaluating whether they fix a vulnerability and generating exploit guidance. That will be the case for most major open source projects (nginx, OpenSSL, Postgres, &c) sooner rather than later.

The norms of coordinated disclosure are not calibrated for this environment. They really haven't been for the last decade.

I'm weirdly comfortable with this, because I think coordinated disclosure norms have always been blinkered, based on the unquestioned premise that delaying disclosure for the operational convenience of system administrators is a good thing. There are reasons to question that premise! The delay also keeps information out of the hands of system operators who have options other than applying patches.

Re: AI is breaking two vulnerability cultures

#42

Post by awesome_dude »

ragall wrote: post #38
awesome_dude wrote: post #37

Earlier quoted context omitted.

> Pragmatically, correlation is evidence of causation in favour of the best explanation, until somebody finds a better explanation. Uh, no. Correlation is only ever one thing - cause for investigation. Everything based on correlation alone is speculation. You can speculate all you like, I have zero issue with that, but that's best prefaced with "I guess" edit: Science captures this perfectly, and people misunderstand…

Very often you only have limited time for investigation and you have to act now . Action is almost always based on educated guesses.

You have moved from "We know" to "We have an educated guess" which is the right way to couch things.

However I wanted to also point out that relying only on educated guesses can lead us into a position where we are "papering over the cracks" or "addressing the symptoms", not the "underlying cause"

Yes, sometimes that's all that can be done, but, also, sometimes it can be more damaging than the cause itself (thinking in terms of the cause continuing to fester away, whilst we think it's 'solved')

Re: AI is breaking two vulnerability cultures

#43

Post by perlgeek »

proteal wrote: post #39

It sounds to me like the safe assumption with software is that no matter how solid your stack is, there are vulnerabilities, potentially catastrophic. A question to folks more experienced than me - if my business depends on software, and I know that my software is almost certainly exploitable, how do I posture my business in such a way as to minimize the impacts of exploits like these?

When Windows was the predominant desktop OS in the 90s and maybe early 00s (ok, maybe still is), it was so badly insecure that you could be pretty much sure that it would be easy to compromise.

That's when firewalls were widely deployed to provide some layer of protection.

So you can ask yourself, what is the (possibly metaphorical) firewall in the software you depend on?

Is there any way you can decrease attack surface, separate out the most important data in extra-secure (and thus less accessible) systems?

Re: AI is breaking two vulnerability cultures

#44

Post by grog454 »

tptacek wrote: post #41

This has been a very long time coming and the crackup we're starting to see was predicted long before anyone knew what an LLM is. The catalyst is the shift towards software transparency: both the radically increased adoption of open source and source-available software, and the radically improved capabilities of reversing and decompilation tools. It has been over a decade since any ordinary off-the-shelf closed-sourc…

> It has been over a decade since any ordinary off-the-shelf closed-source software was meaningfully obscured from serious adversaries.

Probably goes without saying but the last line of defense is not deploying your software publicly and instead relying on server-client architectures to do anything. Maybe this will be more common as vulnerabilities are more easily detected and exploited. Of course its not always feasible.

It has been annoying seeing my (proguard obfuscated) game client binaries decompiled and published on github many times over the last 11 years. Only the undeployed server code has remained private.

Interestingly I didn't have a problem with adversaries reverse engineering my network protocols until I was updating them less frequently than weekly. LLM assisted adversaries could probably keep up with that now too.

Re: AI is breaking two vulnerability cultures

#45

Post by awesome_dude »

tempestn wrote: post #32
awesome_dude wrote: post #24

Earlier quoted context omitted.

That's correlation, not causation. It could equally be argued that the AI slop that's being produced makes for a lot more vulnerabilities being shipped. The bigger target makes for the easier discovery.

But don't we know that some of the vulnerabilities being discovered predate ai coding?

Certainly, and some discoveries have been attributed to AI (I was reading that mozilla firefox were praising mythos recently)

But that's not accounting for all of the discoveries, not at all.

I've also seen the npm people talking about the surge in AI code overwhelming the ability to properly review what's being distributed, and a large number of vulnerabilities being attributed to that

Re: AI is breaking two vulnerability cultures

#46

Post by Animats »

We have a huge problem.

The US is at war. Much of the world is at war at the cyber attack level right now. The US, the EU, most of the Middle East, Israel, Russia... Major services have been attacked and have gone down for days at a time - Ubuntu, Github, Let's Encrypt, Stryker. Entire hospital systems have had to partially shut down.

Now, in the middle of this, AI has made attacks much faster to generate. Faster than the defensive side can respond. Zero-day attacks used to be rare. Now they're normal.

It's going to get worse before it gets better. Maybe much worse.

Re: AI is breaking two vulnerability cultures

#47

Post by cryo32 »

I must admit I'm rather enjoying this particular form of shit show, mostly because it was a predication I made in 2023 in the early days of LLMs. It wasn't really a problem related to LLMs but a glaring hole in the thinking of current computing which is the "frustratingly over-connected" and "over-trust" approach to everything. After reading Liu Cixin's "three body problem" and noting the Dark Forest, I applied that to risk vectors and came to the conclusion that our over-connected nature plus some form of acceleration plus some form of negative impact will fuck us big time.

Turns out it did.

Thus we should probably start treating our thinking model of computing as a Dark Forest, not a friendly community. That mitigates these risks to some degree.

Re: AI is breaking two vulnerability cultures

#48

Post by lynndotpy »

rikafurude21 wrote: post #3

This feels more like an old problem getting reframed as an AI problem. people were already diffing kernel commits and figuring out which ones were security fixes long before llms. if a patch lands publicly, the race has basically already started. also not sure shorter embargoes really help. the orgs that can patch in hours are already fine. everyone else still takes days or weeks. if anything, cheaper exploit generat…

I haven't been keeping tabs for the entirety of Linux development, but has it ever happened before that someone dropped a working exploit from the mailing list before the patch even hit the kernel?

I haven't seen this kind of thing and I get the impression, despite all the hype, that this will be a frequent phenomenon now thanks to LLMs.

Re: AI is breaking two vulnerability cultures

#49

Post by pier25 »

Animats wrote: post #46

We have a huge problem. The US is at war. Much of the world is at war at the cyber attack level right now. The US, the EU, most of the Middle East, Israel, Russia... Major services have been attacked and have gone down for days at a time - Ubuntu, Github, Let's Encrypt, Stryker. Entire hospital systems have had to partially shut down. Now, in the middle of this, AI has made attacks much faster to generate. Faster tha…

> before it gets better

How is it going to get better?

Re: AI is breaking two vulnerability cultures

#50

Post by jefftk »

pier25 wrote: post #49
Animats wrote: post #46

We have a huge problem. The US is at war. Much of the world is at war at the cyber attack level right now. The US, the EU, most of the Middle East, Israel, Russia... Major services have been attacked and have gone down for days at a time - Ubuntu, Github, Let's Encrypt, Stryker. Entire hospital systems have had to partially shut down. Now, in the middle of this, AI has made attacks much faster to generate. Faster tha…

> before it gets better How is it going to get better?

I'd speculate that at this point Linux etc are probably having vulnerabilities discovered and patched faster than created.
Post reply on HN