Live data from Hacker News

AI Slop vs. OSS Security

devansh.bearblog.dev

21–30 of 124 posts

Re: AI Slop vs. OSS Security

#21
post #3

Earlier quoted context omitted.

I am assuming that millions of dollars have already been spent trying to get LLMs to hallucinate less. Even if Problems feel soluble, they often aren't. You might have to invent an entirely new paradigm of text generation to solve the hallucination problem. Or it could be the Collatz Conjecture of LLMs, that it "feels" so possible, but you never really get there.

Nuclear fusion was always 30 years away (c)

It would be nice if nuclear fusion had the AI budget.

Re: AI Slop vs. OSS Security

#22
post #2

> The model has no concept of truth—only of plausibility. This is such an important problem to solve, and it feels soluble. Perhaps a layer with heavily biased weights, trained on carefully curated definitional data. If we could train in a sense of truth - even a small one - many of the hallucinatory patterns disappear. Hats off to the curl maintainers. You are the xkcd jenga block at the base.

Truth comes from being able to test your assertions. Without that they remain in the realm of plausibility. You can't get from plausibility to truth with better training data, you need to give LLMs better tools to test the truth of their plausible statements before spewing them to the user (and train the models to use them, obviously. But that's not the hard part).

Re: AI Slop vs. OSS Security

#23
I don't know why submitting a vulnerability on those platforms is still free. If reporters had to pay a little amount of money (let's say, 20-50$, or indexed to the maximum gain of a vulnerability in a given category) when submitting their report, maybe those would be of better quality.

I know that this poses new problems (some people can't afford to spend this money), but it would be better than just wasting people's time.

Re: AI Slop vs. OSS Security

#24
post #23

I don't know why submitting a vulnerability on those platforms is still free. If reporters had to pay a little amount of money (let's say, 20-50$, or indexed to the maximum gain of a vulnerability in a given category) when submitting their report, maybe those would be of better quality. I know that this poses new problems (some people can't afford to spend this money), but it would be better than just wasting people'…

This point is brought up and discussed in the linked article.

Re: AI Slop vs. OSS Security

#25
I believe "Economic Friction" is truly the best and potentially strongest way to deal with this.

It's good for the site collecting the fee, it's good for the projects being reported on and it doesn't negatively affect valid reports.

It does exactly what we want by disincentivizing bad reports, either AI generated or not.

Re: AI Slop vs. OSS Security

#26

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

I'm so sick of people claiming things sound like AI, when it's so easily not true.

Between this and the flip side of AI-slop it's getting really frustrating out here online.

Re: AI Slop vs. OSS Security

#27
post #20

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!" In point of fact, I had not. After the security reporting issue, the next problem on the list is "trust in other people's writing".

I think one potential downside of using LLMs or exposing yourself to their generated content is that you may subconsciously adopt their quirks over time. Even if you aren't actively using AI for a particular task, prior exposure to their outputs could be biasing your thoughts.

This has additional layers to it as well. For example, I actively avoid using em dash or anything that resembles it right now. If I had no exposure to the drama around AI, I wouldn't even be thinking about this. I am constraining my writing simply to avoid the implication.

Re: AI Slop vs. OSS Security

#28

The solution isn't to block aggressively or to allow everything, but to prioritize. Put accounts older than the AI boom at the top, and allow them to give "referrals", ie stake a part of their own credibility to boost another account on the priority ladder. Referral systems are very efficient at filtering noise.

This is very short sighted.

Re: AI Slop vs. OSS Security

#29
post #27
post #20

Earlier quoted context omitted.

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!" In point of fact, I had not. After the security reporting issue, the next problem on the list is "trust in other people's writing".

I think one potential downside of using LLMs or exposing yourself to their generated content is that you may subconsciously adopt their quirks over time. Even if you aren't actively using AI for a particular task, prior exposure to their outputs could be biasing your thoughts. This has additional layers to it as well. For example, I actively avoid using em dash or anything that resembles it right now. If I had no exp…

Exactly and this is hell for programming.

You don't know whose style the LLM would pick for that particular prompt and project. You might end up with Carmack or maybe that buggy, test-failing piece of junk project on Github.

Re: AI Slop vs. OSS Security

#30
>First, the typical AI-powered reporter, especially one just pasting GPT output into a submission form, neither knows enough about the actual codebase being examined nor understands the security implications well enough to provide insight that projects need.

How ironic, considering every time I've reported a complicated issue to a program on HackerOne, the triggers have completely rejected them because they do not understand the complicated codebase that they are triaging for.

Also the curl examples given in TFA completely ignore recent developments, where curl's maintainers welcomed and fixed literally hundred of AI-found bugs: https://www.theregister.com/2025/10/02/curl_project_swamped_...

Post reply on HN