Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Slop vs. OSS Security

devansh.bearblog.dev

Post reply on HN
Print view View original

51–60 of 124 posts

Re: AI Slop vs. OSS Security

#51

Post by alchemism »

Taek wrote: post #45
code51 wrote: post #29

Earlier quoted context omitted.

Exactly and this is hell for programming. You don't know whose style the LLM would pick for that particular prompt and project. You might end up with Carmack or maybe that buggy, test-failing piece of junk project on Github.

You can tell it who's style to copy, it's actually decent at following instructions like that.

My first experiments with LLM chat was to ask to produce text mimicking the style of a distinct, well-known author. It was also quite good at producing hybrid fusions of unique fictional styles, A + B = AB.

Re: AI Slop vs. OSS Security

#52

Post by bklosky »

Companies soliciting big bounties should charge a fee to submit, making slop costly. Steam, the video game publisher, does this. Developers need to pay a fee to list their game, and if their game sells sufficient volume, the fee is returned. Creating a separating equilibrium here is not hard, the hand wringing is weird to me.

Re: AI Slop vs. OSS Security

#53

Post by dec0dedab0de »

bluGill wrote: post #49
nixpulvis wrote: post #26

Earlier quoted context omitted.

I'm so sick of people claiming things sound like AI, when it's so easily not true. Between this and the flip side of AI-slop it's getting really frustrating out here online.

Problem is AI sounds like normal people. There are a few clues, but good writers have always sounded like AI - except good writers are not making things up.

AI text often sounds like Corporate/PR nonsense to me. I was already convinced that people speaking like that were robots 30 years ago.

Re: AI Slop vs. OSS Security

#54

Post by ndiddy »

Cthulhu_ wrote: post #34
supriyo-biswas wrote: post #19

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

Using the word is implies you have definite, conclusive proof, but the only one is a number of phrases that you believe are tells for AI generated stuff, but is it really or are you only now paying extra attention to it? It's better to stay neutral and say you suspect it may be AI generated. And for everyone else, responsible disclosure of using AI tools to write stuff would be appreciated. (this comment did not invo…

> Using the word is implies you have definite, conclusive proof, but the only one is a number of phrases that you believe are tells for AI generated stuff, but is it really or are you only now paying extra attention to it? It's better to stay neutral and say you suspect it may be AI generated.

Literally the first two sentences on the linked article:

> Disclosure: Certain sections of this content were grammatically refined/updated using AI assistance, as English is not my first language. Quite ironic, I know, given the subject being discussed.

Personally, I've read enough AI generated SEO spam that anything with the AI voice comes off as being inauthentic and spammy. I would much rather read something with the mistakes a non-native English speaker would make than something AI written/edited.

Re: AI Slop vs. OSS Security

#55

Post by theptip »

It seems pretty obvious that the bar needs to be raised.

> A security report lands in your inbox. It claims there's a buffer overflow in a specific function. The report is well-formatted, includes CVE-style nomenclature, and uses appropriate technical language.

Given how easy it is to generate a POC these days, I wonder if HackerOne needs to be pivoting hard into scaffolding to help bug hunters prove their vulns.

- Claude skills/MCP for OSS projects

- Attested logging/monitoring for API investigations (eg hosted BURP)

Re: AI Slop vs. OSS Security

#56

Post by wahnfrieden »

noirscape wrote: post #11
dvt wrote: post #6

> Requiring technical evidence such as screencasts showing reproducibility, integration or unit tests demonstrating the fault, or complete reproduction steps with logs and source code makes it much harder to submit slop. If this isn't already a requirement, I'm not sure I understand what even non-AI-generated reports look like. Isn't the bare-minimum of CVE reporting a minimally reproducible example? Like, even if yo…

The problem that is that a lot of CVEs often don't represent "real" vulnerabilities, but merely theoretical ones that could hypothetically be combined to make a real exploit. Regex exploitation is the forever example to bring up here, as it's generally the main reason that "autofail the CI system the moment an auditing command fails" doesn't work on certain codebases. The reason this happens is because it's trivial t…

Aren't there regex libraries that aren't susceptible to that

Re: AI Slop vs. OSS Security

#57

Post by __zack »

Cthulhu_ wrote: post #34
supriyo-biswas wrote: post #19

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

Using the word is implies you have definite, conclusive proof, but the only one is a number of phrases that you believe are tells for AI generated stuff, but is it really or are you only now paying extra attention to it? It's better to stay neutral and say you suspect it may be AI generated. And for everyone else, responsible disclosure of using AI tools to write stuff would be appreciated. (this comment did not invo…

It looks like the author did in fact update their article to disclose AI use:

> Certain sections of this content were grammatically refined/updated using AI assistance

> I don't know how to write an emdash

Same here, and at this point I don’t think I will ever learn

Re: AI Slop vs. OSS Security

#58

Post by dafelst »

Sharlin wrote: post #47
goalieca wrote: post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.

I agree wholeheartedly, and this is the core problem - many of the people evangelizing LLMs for a particular task (especially investors and AI gold rush "entrepreneurs") do not have enough expertise in that particular field to effectively evaluate the quality of the output. It sure looks the part though, and for those with a shallow understanding, it is often enough.

Re: AI Slop vs. OSS Security

#59

Post by riskable »

thadt wrote: post #20
supriyo-biswas wrote: post #19

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!" In point of fact, I had not. After the security reporting issue, the next problem on the list is "trust in other people's writing".

Exposure to AI leads to people writing like AI. Just like when you're hanging out in certain circles, you start to talk like those people. It's human nature.

Re: AI Slop vs. OSS Security

#60

Post by riskable »

bob1029 wrote: post #27
thadt wrote: post #20

Earlier quoted context omitted.

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!" In point of fact, I had not. After the security reporting issue, the next problem on the list is "trust in other people's writing".

I think one potential downside of using LLMs or exposing yourself to their generated content is that you may subconsciously adopt their quirks over time. Even if you aren't actively using AI for a particular task, prior exposure to their outputs could be biasing your thoughts. This has additional layers to it as well. For example, I actively avoid using em dash or anything that resembles it right now. If I had no exp…

I suddenly have the urge to reply to this with a bulleted list where the bullets are emoji.
Post reply on HN