Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite Critical CVEs or LLM Slop?

research.jfrog.com

Post reply on HN
Print view View original

151–160 of 407 posts

Re: SQLite Critical CVEs or LLM Slop?

#151

Post by gortok »

red75prime wrote: post #139
gortok wrote: post #52

We can chalk this up as another example of over-exhuberance by what folks believe LLMs can accomplish vs. what they actually are. LLM-based “AI” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credi…

Apophatic intelligence? "We don't know what intelligence is, but LLMs with CoT are certainly not it despite being Turing-complete." Watching for unexpected failure modes is surely worth it.

Turing-completeness is a necessary pre-requisite for being able to fulfill the requirements of a Turing machine, nothing more. In the same way that cell division is a necessary condition for life, but cell division does not mean a given life form itself is sentient.

Intelligent life-forms can generate probabilistic outputs based on inputs, but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.

Re: SQLite Critical CVEs or LLM Slop?

#152

Post by thayne »

bluGill wrote: post #53
flerchin wrote: post #39

Earlier quoted context omitted.

The vast majority of CVEs are not exploitable, basically noise. I suspect that the overwhelming majority of the CVEs being generated by LLMs are either noise of the sort in the linked article or noise of the sort that is not exploitable.

The majority these days seems to be real issues, but probably not exploitable. The latest LLMs are very good at reading code and finding issues where the code is wrong. They are somewhat less good at finding exploits - both because finding exploits is harder than finding code that is clearly wrong (even if not exploitable), and also because the better ones are intentionally made to not create exploits (since they don…

> I suspect the repo in question was generated with a year-old LLM

I suspect there are a lot of people running inexpensive models that are searching for vulnerabilities across a lot of projects, probably in an automated way (ex with openclaw or similar) in the hopes of winning a bounty and/or noteriety.

Re: SQLite Critical CVEs or LLM Slop?

#153

Post by bluGill »

mwkaufma wrote: post #142
bluGill wrote: post #72

Earlier quoted context omitted.

The real question is will [when will] these vulnerabilities get fixed faster than they can be found. LLMs are finding a lot of real bugs that have existed without being noticed for 20+ years. It is generally expected [hoped] that in a short time all those 20 year old bugs will be fixed and LLMs will settle down to just things introduced in the last month (or perhaps this pull request) and thus be manageable.

Won't an LLM, prompted for a CVE report, always hallucinate one even if it isn't there? Why would they "settle down"?

Two different points.

First, is there any reason they have to? The LLM makers are aware of this problem and doing things to stop it. Maybe they will start saying "I can't find anything". Already they refuse to provide some exploits.

Second, most people are not prompting for a CVE, they are asking to review and find issues if any. Which sometimes finds a CVE, but in my experience it sometimes does say "looks good".

Re: SQLite Critical CVEs or LLM Slop?

#154

Post by clbrmbr »

dwedge wrote: post #47
clbrmbr wrote: post #13

Earlier quoted context omitted.

Many orgs (esp w ISO27000) have a vulnerability management policy that involves patching at least critical CVEs within a short timeline. Tools like trivvy make it possible to do the scans…

> Tools like trivvy make it possible to do the scans... Only if you didn't rip trivvy out of your organisation when it had two supply chain compromises within a month of each other earlier this year

Yikes. Man, there’s a market opening for someone to redistribute open source projects with supply chain assurances!

Re: SQLite Critical CVEs or LLM Slop?

#155

Post by bluGill »

thayne wrote: post #152
bluGill wrote: post #53

Earlier quoted context omitted.

The majority these days seems to be real issues, but probably not exploitable. The latest LLMs are very good at reading code and finding issues where the code is wrong. They are somewhat less good at finding exploits - both because finding exploits is harder than finding code that is clearly wrong (even if not exploitable), and also because the better ones are intentionally made to not create exploits (since they don…

> I suspect the repo in question was generated with a year-old LLM I suspect there are a lot of people running inexpensive models that are searching for vulnerabilities across a lot of projects, probably in an automated way (ex with openclaw or similar) in the hopes of winning a bounty and/or noteriety.

Most bounty programs are aware of this. Many projects have closed their bounty program in response to that type of attack.

Re: SQLite Critical CVEs or LLM Slop?

#156

Post by rowls66 »

MattPalmer1086 wrote: post #104
jodacola wrote: post #85

Earlier quoted context omitted.

I'm sending all my condolences out to everyone who is in a corporate environment using tools like Veracode or Snyk to automatically assess for CVEs and block merges or deployments based on "findings". My experience in such environments leads me to believe this is going to be a rough ride for those heavily locked-down enterprises, because depending on the environment, an exception of "this CVE was hallucinated by AI"…

In our organisation, my security team spend the majority of our effort weeding out unexploitable CVEs - which are the majority of them. But the volume is growing year on year - I don't know how sustainable it's going to be.

Sounds like a good job for AI. I do fear that the future will involve deploying AI do deal with the mountain of slop produced by other AI's.

Re: SQLite Critical CVEs or LLM Slop?

#158

Post by VCFundedGenYer »

ChrisMarshallNY wrote: post #4

The problem with this kind of thing, is that it reduces the S/N (Signal-to-Noise) ratio, so weeding out the legit CVEs becomes a lot more difficult. But, on the other hand, I do know that LLMs have been discovering a lot of legit CVEs, and I will lay odds that the blackhats are leveraging them to the max.

Which is precisely why I do not celebrate when Microsoft or other companies say they released "their biggest set of security vulnerability fixes ever" because there's a good chance that most of them are hallucinations or non-critical issues erroneously considered zero-days or critical.

What actually needs to be done is human review, vetting, and approval of those. Otherwise, it's just unknown/unnecessary/potentially dangerous changesets in high quantities.

Re: SQLite Critical CVEs or LLM Slop?

#159

Post by RetpolineDrama »

flerchin wrote: post #39
ChrisMarshallNY wrote: post #4

The problem with this kind of thing, is that it reduces the S/N (Signal-to-Noise) ratio, so weeding out the legit CVEs becomes a lot more difficult. But, on the other hand, I do know that LLMs have been discovering a lot of legit CVEs, and I will lay odds that the blackhats are leveraging them to the max.

The vast majority of CVEs are not exploitable, basically noise. I suspect that the overwhelming majority of the CVEs being generated by LLMs are either noise of the sort in the linked article or noise of the sort that is not exploitable.

>The vast majority of CVEs are not exploitable

Perhaps in isolation. The issue is when you can chain exploits to bypass multiple layers.

Post reply on HN