AI Slop vs. OSS Security • Hacker News Board

Re: AI Slop vs. OSS Security

#41

Post by mschuster91 » Thu, Nov 06, 2025, 2:45 PM UTC

samlinnfer wrote: post #32

Just add a country IP ban, we all know who is submitting these reports. Remember Hacktoberfest?

> Just add a country IP ban, we all know who is submitting these reports.

As much as I'd like to see Russia, China and India disconnected off of the wide Internet until they clean up shop with abusive actors, the Hacktoberfest stuff you're likely referring to doesn't have anything to do with your implication - that was just a chance at a free t-shirt [1] that caused all the noise.

In ye olde times, you'd need to take care how you behaved in public because pulling off a stunt like that could reasonably lead to your company going out of business - but even a "small" company like DO is too big to fail from FAFO, much less ultra large corporations like Google that just run on sheer moat. IMHO, that is where we have to start - break up the giants, maybe that's enough of a warning signal to also alert "smaller" large companies to behave like citizens again.

[1] https://domenic.me/hacktoberfest/

Re: AI Slop vs. OSS Security

#42

Post by progbits » Thu, Nov 06, 2025, 2:46 PM UTC

nixpulvis wrote: post #26

supriyo-biswas wrote: post #19

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

I'm so sick of people claiming things sound like AI, when it's so easily not true. Between this and the flip side of AI-slop it's getting really frustrating out here online.

From the article itself (presumably added later):

  Disclosure: Certain sections of this content were grammatically refined/updated using AI assistance

Re: AI Slop vs. OSS Security

#43

Post by progbits » Thu, Nov 06, 2025, 2:47 PM UTC

cheesecompiler wrote: post #38

supriyo-biswas wrote: post #19

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

My least favourite part of this timeline: anyone who writes well gets classified as AI. Some of us press Option+- to insert an em dash and have been for years.

GP did not use emdash as evidence of AI but rather the verbose style with little signal/noise ratio and the typical phrases most LLMs like to use.

Re: AI Slop vs. OSS Security

#44

Post by progbits » Thu, Nov 06, 2025, 2:51 PM UTC

  Certain sections of this content were grammatically refined/updated using AI assistance, as English is not my first language.

OP: I sympathize, but I would much rather read your original text, with typos and grammatical errors. By feeding it through the LLM you fix issues that are not really important but remove your own voice and get a bland slop identical to 90% of these slopblogs (which your's isn't!)

Re: AI Slop vs. OSS Security

#45

Post by Taek » Thu, Nov 06, 2025, 2:56 PM UTC

code51 wrote: post #29

bob1029 wrote: post #27

Earlier quoted context omitted.

I think one potential downside of using LLMs or exposing yourself to their generated content is that you may subconsciously adopt their quirks over time. Even if you aren't actively using AI for a particular task, prior exposure to their outputs could be biasing your thoughts. This has additional layers to it as well. For example, I actively avoid using em dash or anything that resembles it right now. If I had no exp…

Exactly and this is hell for programming. You don't know whose style the LLM would pick for that particular prompt and project. You might end up with Carmack or maybe that buggy, test-failing piece of junk project on Github.

You can tell it who's style to copy, it's actually decent at following instructions like that.

Re: AI Slop vs. OSS Security

#46

Post by danieltk76 » Thu, Nov 06, 2025, 3:00 PM UTC

At Vulnetic ai we deal with this by having a separate validator agent attack the vulnerability reported using a novel perspective separate of the hacking agent.

Re: AI Slop vs. OSS Security

#47

Post by Sharlin » Thu, Nov 06, 2025, 3:05 PM UTC

goalieca wrote: post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.

Re: AI Slop vs. OSS Security

#48

Post by Sharlin » Thu, Nov 06, 2025, 3:06 PM UTC

> The incentive is to submit as many reports as possible and see what sticks, because even a 5% hit rate on a hundred submissions is better than the effort of manually verifying five findings.

As I just commented in the other AI trust thread on the front page, this dynamic is funnily enough what any woman using online dating services has always been very familiar with. With the exact same tragedy of the commons that results. Except for the important difference that terrible profiles and intro messages have traditionally usually been very short and easily red-flagged. But that is, of course, now also changing or already changed due to LLMs.

(Someone I follow on a certain social media platform just remarked that she got no less than fifty messages within a single hour of marking herself as "single". And she's just some average person, not a "star" of any sort.)

Re: AI Slop vs. OSS Security

#49

Post by bluGill » Thu, Nov 06, 2025, 3:08 PM UTC

nixpulvis wrote: post #26

supriyo-biswas wrote: post #19

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

I'm so sick of people claiming things sound like AI, when it's so easily not true. Between this and the flip side of AI-slop it's getting really frustrating out here online.

Problem is AI sounds like normal people. There are a few clues, but good writers have always sounded like AI - except good writers are not making things up.

Re: AI Slop vs. OSS Security

#50

Post by samuelknight » Thu, Nov 06, 2025, 3:11 PM UTC

You can address the issue by putting the report and the code base in a sandbox with an agent that tries to reproduce it. If it can't reproduce it then that should be a strike against the reporter. OSS projects should absolutely ban accounts that repetitively create reports that are of such low quality that it can't be recreated. IMO the Hacker One reputation mechanism is a good idea because it incentives users who operate in good faith and can serially produce findings.