AI Slop vs. OSS Security • Hacker News Board

Re: AI Slop vs. OSS Security

#11

Post by noirscape » Thu, Nov 06, 2025, 1:15 PM UTC

dvt wrote: post #6

> Requiring technical evidence such as screencasts showing reproducibility, integration or unit tests demonstrating the fault, or complete reproduction steps with logs and source code makes it much harder to submit slop. If this isn't already a requirement, I'm not sure I understand what even non-AI-generated reports look like. Isn't the bare-minimum of CVE reporting a minimally reproducible example? Like, even if yo…

The problem that is that a lot of CVEs often don't represent "real" vulnerabilities, but merely theoretical ones that could hypothetically be combined to make a real exploit.

Regex exploitation is the forever example to bring up here, as it's generally the main reason that "autofail the CI system the moment an auditing command fails" doesn't work on certain codebases. The reason this happens is because it's trivial to make a string that can waste significant resources to try and do a regex match against, and the moment you have a function that accepts a user-supplied regex pattern, that's suddenly an exploit... which gets a CVE. A lot of projects then have CVEs filed against them because internal functions rely on Regex calls as arguments, even if they're in code the user is flat-out never going to be able interact with (ie. Several dozen layers deep in framework soup there's a regex call somewhere, in a way the user won't be able to access unless a developer several layers up starts breaking the framework they're using in really weird ways on purpose).

The CVE system is just completely broken and barely serves as an indicator of much of anything really. The approval system from what I can tell favors acceptance over rejection, since the people reviewing the initial CVE filing aren't the same people that actively investigate if the CVE is bogus or not and the incentive for the CVE system is literally to encourage companies to give a shit about software security (at the same time, this fact is also often exploited to create beg bounties). CVEs have been filed against software for what amounts to "a computer allows a user to do things on it" even before AI slop made everything worse; the system was questionable in quality 7 years ago at the very least, and is even worse these days.

The only indicator it really gives is that a real security exploit can feel more legitimate if it gets a CVE assigned to it.

Re: AI Slop vs. OSS Security

#12

Post by beeburrt » Thu, Nov 06, 2025, 1:20 PM UTC

I didn't realize how bleak the future looks, wrt CVE infastructure both MITRE and at National Vuln. Database.

What do other countries do for their stuff like this?

Re: AI Slop vs. OSS Security

#13

Post by jsheard » Thu, Nov 06, 2025, 1:22 PM UTC

goalieca wrote: post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

The other fundamental problem is that to a grifter, it's not a fundamental problem for the output to be plausible but often wrong. Plausible is all they need.

Re: AI Slop vs. OSS Security

#14

Post by bawolff » Thu, Nov 06, 2025, 1:28 PM UTC

dvt wrote: post #6

> Requiring technical evidence such as screencasts showing reproducibility, integration or unit tests demonstrating the fault, or complete reproduction steps with logs and source code makes it much harder to submit slop. If this isn't already a requirement, I'm not sure I understand what even non-AI-generated reports look like. Isn't the bare-minimum of CVE reporting a minimally reproducible example? Like, even if yo…

As someone who worked on the recieving end of security reports, often not. They can be surprisingly poorly written.

You sort of want to reject them all, but ocassionally a gem gets submitted which makes you reluctant.

For example, years ago i was responsible for triaging bug bounty reports at a SaaS company i worked at at the time. One of the most interesting reports was that someone found a way to bypass our oauth thing by using a bug in safari that allowed them to bypass most oauth forms. The report was barely understandable written in broken english. The impression i got was they tried to send it to apple but apple ignored them. We ended up rewriting the report and submitting it to apple on there behalf (we made sure the reporter got all credit).

If we ignored poorly written reports we would have missed that. Is it worth it though? I dont know.

Re: AI Slop vs. OSS Security

#15

Post by Maro » Thu, Nov 06, 2025, 1:31 PM UTC

Manufacturing vulnerability submissions that look like real vulnerability submissions, but the vulnerability isn't there and the submitter doesn't understand what it's saying.

It's a cargo cult. Maybe the airplanes will land and bring the goodies!

https://en.wikipedia.org/wiki/Cargo_cult

Re: AI Slop vs. OSS Security

#16

Post by hshdhdhehd » Thu, Nov 06, 2025, 1:32 PM UTC

bawolff wrote: post #14

dvt wrote: post #6

> Requiring technical evidence such as screencasts showing reproducibility, integration or unit tests demonstrating the fault, or complete reproduction steps with logs and source code makes it much harder to submit slop. If this isn't already a requirement, I'm not sure I understand what even non-AI-generated reports look like. Isn't the bare-minimum of CVE reporting a minimally reproducible example? Like, even if yo…

As someone who worked on the recieving end of security reports, often not. They can be surprisingly poorly written. You sort of want to reject them all, but ocassionally a gem gets submitted which makes you reluctant. For example, years ago i was responsible for triaging bug bounty reports at a SaaS company i worked at at the time. One of the most interesting reports was that someone found a way to bypass our oauth t…

In the AI age I'd prefer poorly written reports in broken English. Just as long as that doesnt become a known bypass and so the AI is instructed to sound broken.

Re: AI Slop vs. OSS Security

#17

Post by skydhash » Thu, Nov 06, 2025, 1:32 PM UTC

Jean-Papoulos wrote: post #8

The solution isn't to block aggressively or to allow everything, but to prioritize. Put accounts older than the AI boom at the top, and allow them to give "referrals", ie stake a part of their own credibility to boost another account on the priority ladder. Referral systems are very efficient at filtering noise.

It only works when it’s a true stake. Like loosing privileges when a referral is a dunce. The downside is tribalism.

Re: AI Slop vs. OSS Security

#18

Post by wwfn » Thu, Nov 06, 2025, 1:43 PM UTC

big-and-small wrote: post #5

wwfn wrote: post #4

Wealth generated on top of underpaid labor is a reoccurring theme -- and in this case maybe surprisingly exacerbated by LLMs. Would this be different if the underlying code had a viral license? If google's infrastructure was built on a GPL'ed libcurl [0], would they have investment in the code/a team with resources to evaluate security reports (slop or otherwise)? Ditto for libxml. Does GPL help the linux kernel get…

Copyleft licenses are made to support freedom for everyone and particularly end-users. They only limit freedom of developers / maintainers to exploit the code and users. > Does GPL help the linux kernel get investment from it's corporate users? GPL has helped "linux kernel the project" greatly, but companies invest in it out of their self-interest. They want to benefit from upstream improvements and playing nicely by…

I would have thought supporting libcurl and libxml would also be in a company's self-interest. Is that companies do this for GPL'ed linux kernel but not BSD evidence that strong copyleft licensing limits the extent to which OSS projects are exploited/under-resourced?

Re: AI Slop vs. OSS Security

#19

Post by supriyo-biswas » Thu, Nov 06, 2025, 1:45 PM UTC

Ironically, even this piece is significantly AI-generated:

- Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs

- "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing.

- The listicle under "What Might Actually Work"

Re: AI Slop vs. OSS Security

#20

Post by thadt » Thu, Nov 06, 2025, 1:54 PM UTC

supriyo-biswas wrote: post #19

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!"

In point of fact, I had not.

After the security reporting issue, the next problem on the list is "trust in other people's writing".