Live data from Hacker News

AI Slop vs. OSS Security

devansh.bearblog.dev

71–80 of 124 posts

Re: AI Slop vs. OSS Security

#71

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

[deleted]

Re: AI Slop vs. OSS Security

#72
post #27
post #20

Earlier quoted context omitted.

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!" In point of fact, I had not. After the security reporting issue, the next problem on the list is "trust in other people's writing".

I think one potential downside of using LLMs or exposing yourself to their generated content is that you may subconsciously adopt their quirks over time. Even if you aren't actively using AI for a particular task, prior exposure to their outputs could be biasing your thoughts. This has additional layers to it as well. For example, I actively avoid using em dash or anything that resembles it right now. If I had no exp…

Out of the mountains of content, one single symbol would provoke the ire of non-ascii reactionaries.

https://news.ycombinator.com/item?id=44072922

https://news.ycombinator.com/item?id=45766969

https://news.ycombinator.com/item?id=45073287

Re: AI Slop vs. OSS Security

#73
post #27
post #20

Earlier quoted context omitted.

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!" In point of fact, I had not. After the security reporting issue, the next problem on the list is "trust in other people's writing".

I think one potential downside of using LLMs or exposing yourself to their generated content is that you may subconsciously adopt their quirks over time. Even if you aren't actively using AI for a particular task, prior exposure to their outputs could be biasing your thoughts. This has additional layers to it as well. For example, I actively avoid using em dash or anything that resembles it right now. If I had no exp…

Isn't the alternative far more likely? These tools were trained on the way people write in certain settings, which includes a lot of curated technical articles like this one, and we're seeing that echoed in their output.

There's no "LLM style". There's "human style mimicked by LLMs". If they default to a specific style, then that's on the human user who chooses to go with it, or, likely, doesn't care. They could just as well make it output text in the style of Shakespeare or a pirate, eschew emojis and bulleted lists, etc.

If you're finding yourself influenced by LLMs—don't be. Here's why:

• It doesn't matter.

• Keep whatever style you had before LLMs.

:tada:

Re: AI Slop vs. OSS Security

#74
post #20

Ironically, even this piece is significantly AI-generated: - Primarily relies on a single piece of evidence from the curl project, and expands it into multiple paragraphs - "But here's the gut punch:", "You're not building ... You're addressing ...", "This is the fundamental problem:" and so many other instances of Linkedin-esque writing. - The listicle under "What Might Actually Work"

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!" In point of fact, I had not. After the security reporting issue, the next problem on the list is "trust in other people's writing".

Already a big problem in art, people go on witch hunt over what they think are signs of AI use.

It's sad because people that are ok with AI art are still enjoying the human art just the same. Somehow their visceral hate of AI-art managed to ruin human art for themselves as well.

Re: AI Slop vs. OSS Security

#75
Require a docker/script that provides for the necessary conditions for the exploit, along with a POC. If something is impossible to provide a POC for, as it's more of a speculative attack, require vetting like arxiv.

Most people's initial contributions are going to be more concrete exploits.

Re: AI Slop vs. OSS Security

#76
Surely there can be a workflow created to "fight fire with fire" and have an AI that reads reports, trained on the code base with explicit instructions to verify all of the telltale signs of slop...? If AI services can handle the nightmare of parsing emails and understanding the psychology of phishing, I am optimistic it can be done for OSS reports.

It doesn't have to make the final judgement, just some sort of filter that automatically flags things like function calls that don't exist in the code.

Re: AI Slop vs. OSS Security

#77
post #20

Earlier quoted context omitted.

Yesterday my wife burst into my office: "You used AI to generate that (podcast) episode summary, we don't sound like that!" In point of fact, I had not. After the security reporting issue, the next problem on the list is "trust in other people's writing".

I blogged about this fundamental demolition of trust a few months ago. HN discussed it here https://news.ycombinator.com/item?id=44384610 The responses were a surprisingly mixed bag. What I thought was a very common sense observation had some heavy detractors in those threads.

You're on a forum full of people trying to profit from this tech. In that context the pushback is obvious.

Re: AI Slop vs. OSS Security

#78
post #13
post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

The other fundamental problem is that to a grifter, it's not a fundamental problem for the output to be plausible but often wrong. Plausible is all they need.

Indeed. The elderly in my family are seeing a substantial uptick of AI generated stuff that looks extremely plausible. Fortunately they're old but not stupid, so far nobody has fallen for any of these but I have to admit: they look good enough to pass a first casual inspection.

Re: AI Slop vs. OSS Security

#79
post #47
post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.

That, combined with the confidence any of its output is communicated back to the user.

Re: AI Slop vs. OSS Security

#80
post #47
post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.

Except for things they happen to know something about.
Post reply on HN