Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Slop vs. OSS Security

devansh.bearblog.dev

Post reply on HN
Print view View original

111–120 of 124 posts

Re: AI Slop vs. OSS Security

#111

Post by spwa4 »

goalieca wrote: post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

But the problem is fundamentally slop, and grifters. It is possible to generate valid security bug reports with an AI agent, and there certainly is value in that. I'd even say a lot of value.

But of course producing fake ones is far easier and cheaper.

Re: AI Slop vs. OSS Security

#112

Post by bluGill »

dec0dedab0de wrote: post #53
bluGill wrote: post #49

Earlier quoted context omitted.

Problem is AI sounds like normal people. There are a few clues, but good writers have always sounded like AI - except good writers are not making things up.

AI text often sounds like Corporate/PR nonsense to me. I was already convinced that people speaking like that were robots 30 years ago.

True, but at least 30 years ago once you filtered through you could get what you needed.

Today there are scams that look just like real companies trying to get you to buy from them instead. Who knows what happens if you put your money down. (Scams were of course always a problem, but there is much less cost to create a scam)

Re: AI Slop vs. OSS Security

#113

Post by 8note »

lukev wrote: post #108
goalieca wrote: post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

"Form without substance" is quite possibly the most accurate way to describe LLMs in general. Extremely powerful (and extremely useful) pattern finders and replicators... and little else.

its certianly why they pair well with tool calls - substance without form

Re: AI Slop vs. OSS Security

#114

Post by 8note »

aidenn0 wrote: post #88
stackedinserter wrote: post #82

So use AI to check AI reports, problem solved.

I cannot tell if you are being serious or sarcastic.

i dunno about problem solved, but id expect to be able to put together an agent that helps requesters format their reports well, and that could run the reproduction steps in a sandbox.

Re: AI Slop vs. OSS Security

#115

Post by 8note »

bklosky wrote: post #52

Companies soliciting big bounties should charge a fee to submit, making slop costly. Steam, the video game publisher, does this. Developers need to pay a fee to list their game, and if their game sells sufficient volume, the fee is returned. Creating a separating equilibrium here is not hard, the hand wringing is weird to me.

adding cost to accounts is handy, but its not like that has solved steam's scam/slop problems.

fake games by fake studios played by fake players is still a thing

Re: AI Slop vs. OSS Security

#116

Post by [deleted] »

gdulli wrote: post #98
jimbokun wrote: post #80

Earlier quoted context omitted.

Except for things they happen to know something about.

Unfortunately, too few people are making the obvious leap from "LLMs aren't great for topics I have expertise in" to "maybe that means LLMs aren't actually great for the other topics either."

[deleted]

Re: AI Slop vs. OSS Security

#117

Post by taneq »

code51 wrote: post #29
bob1029 wrote: post #27

Earlier quoted context omitted.

I think one potential downside of using LLMs or exposing yourself to their generated content is that you may subconsciously adopt their quirks over time. Even if you aren't actively using AI for a particular task, prior exposure to their outputs could be biasing your thoughts. This has additional layers to it as well. For example, I actively avoid using em dash or anything that resembles it right now. If I had no exp…

Exactly and this is hell for programming. You don't know whose style the LLM would pick for that particular prompt and project. You might end up with Carmack or maybe that buggy, test-failing piece of junk project on Github.

Can you just tell it it’s Carmack? :P

Re: AI Slop vs. OSS Security

#118

Post by acchow »

Sharlin wrote: post #47
goalieca wrote: post #9

> This is the fundamental problem: AI can generate the form of security research without the substance. I think this is the fundamental problem of LLMs in general. Some of the time looks just enough right to seem legitimate. Luckily the rest of the time it doesn’t.

Unfortunately, to a majority of the population approximately 100% of LLM output seems entirely legitimate.

I’ve been trying ChatGPT for transit directions on Shanghai’s metro and it has been absolutely terrible. Hallucinating connections and routes.

But all of it’s responses definitely seem convincing (as it has been trained to do)

Re: AI Slop vs. OSS Security

#119

Post by ziml77 »

jerf wrote: post #83
bob1029 wrote: post #27

Earlier quoted context omitted.

I think one potential downside of using LLMs or exposing yourself to their generated content is that you may subconsciously adopt their quirks over time. Even if you aren't actively using AI for a particular task, prior exposure to their outputs could be biasing your thoughts. This has additional layers to it as well. For example, I actively avoid using em dash or anything that resembles it right now. If I had no exp…

I didn't make heavy use of it, but I did sometimes use "It's not X, it's Y" or some closely related variant. I've had to strike that from my writing, because whether or not it makes anyone else cringe, it's making me cringe now. My usage doesn't even match the ones the LLMs favor, my X & Y were typically full clauses with many words rather than the LLM's use of short, punchy X & Ys... but still. Close enough. Can't w…

I wasn't too averse to using the construction until I saw the latest Eddy Burback video where he had an AI encouraging the delusions he was faking for the sake of the video. The sheer number of times the AI said "it's not just X, it's Y" absolutely broke any willingness I had to ever say anything like that again.

Re: AI Slop vs. OSS Security

#120

Post by bawolff »

fisf wrote: post #87
bawolff wrote: post #14

Earlier quoted context omitted.

As someone who worked on the recieving end of security reports, often not. They can be surprisingly poorly written. You sort of want to reject them all, but ocassionally a gem gets submitted which makes you reluctant. For example, years ago i was responsible for triaging bug bounty reports at a SaaS company i worked at at the time. One of the most interesting reports was that someone found a way to bypass our oauth t…

I do not understand. If auth is bypassable, this is not a browser issue, right?

It was a long time ago so i might be misremembering, but i think the idea was that safari would leak the target of redirects cross domain, which allowed the attacker to capture some of the oauth tokens.

So safari was not following the web browser specs in a way that compromised oauth in a common mode of implementation.

Post reply on HN