Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl: We still have not seen a valid security report done with AI help

linkedin.com

Post reply on HN
Print view View original

231–240 of 258 posts

Re: Curl: We still have not seen a valid security report done with AI help

#231

Post by Jean-Papoulos »

holuponemoment wrote: post #185
VladVladikoff wrote: post #119

Earlier quoted context omitted.

This sounds more like an influx of scammers than security researchers leaning too hard on AI tools. The main problem is the bounty structure. And I don’t think these influx of low quality reports will go away, or even get any less aggressive as long as there is money to attract the scammers. Perhaps these bug bounty programs need to develop an automatic pass/fail tester of all submitted bug code, to ensure the report…

Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.

No need to make it a percentage ; charge $1 and the spammers will stop extremely quickly, since none of their reports are valid.

But I do think established individual and institutes should have free access ; leave a choice between going through an identification process and paying the fee. If it's such a big problem that you REALLY need to do something ; otherwise just keep marking as spam.

Re: Curl: We still have not seen a valid security report done with AI help

#232

Post by bloppe »

cedws wrote: post #229
holuponemoment wrote: post #185

Earlier quoted context omitted.

Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.

If you charge a fee the motivation for good samaritan reports goes to zero.

That's why they offer cash bounties. You don't need to charge a fee if there is no bounty (aka an actual good Samaritan situation), cuz then there's no incentive to flood it with slop

Re: Curl: We still have not seen a valid security report done with AI help

#233

Post by Snacklive »

imtringued wrote: post #221
holuponemoment wrote: post #185

Earlier quoted context omitted.

Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.

Why charge a fee? All you need is a reputation system where low reputation bounty hunters need a reputable person to vouch for them. If it turns out to be false, both take a hit. If true, the voucher gets to be a co-author and a share in the bounty.

That's just a way to create a toxic environment filled with elitism similar to StackOverflow

Re: Curl: We still have not seen a valid security report done with AI help

#234

Post by tharant »

namaria wrote: post #225
tharant wrote: post #199

Earlier quoted context omitted.

In my experience, and for use-cases that are carefully considered, language models are not confidently wrong a majority of the time. The trick is understanding the tool and using it appropriately—thus the “carefully considered” approach to identifying use-cases that can provide value.

In the very narrow fields where I have a deep understanding, LLM output is mostly garbage. It sounds plausible but doesn't stand up to scrutiny. The basics that it can regurgitate from wikipedia sound mostly fine but they are already subtly wrong as soon as they depart from stating very basic facts. Thus I have to assume that for any topic I do not fully understand - which is the vast majority of human knowledge - it…

> In the very narrow fields where I have a deep understanding, LLM output is mostly garbage > Thus I have to assume that for any topic I do not fully understand - which is the vast majority of human knowledge - it is worse than useless, it is actively misleading.

Why do you have to make that assumption? An expert arborist likely won’t know much about tuning GC parameters for the JVM but that won’t make them “worse than useless” or “actively misleading” when discussing other topics, and especially not when it comes to the stuff that’s relatively tangential to their domain.

I think the difference we have is that I don’t expect the models to be experts in any domain nor do I expect them to always provide factual content; the library can provide factual content—if you know how to use it right.

Re: Curl: We still have not seen a valid security report done with AI help

#235

Post by rwmj »

VladVladikoff wrote: post #119
danielvf wrote: post #101

I handle reports for a one million dollar bug bounty program. AI spam is bad. We've also never had a valid report from an by an LLM (that we could tell). People using them will take any being told why a bug report is not valid, questions, or asks for clarification and run them back through the same confused LLM. The second pass through generates even deeper nonsense. It's making even responding with anything but "clo…

This sounds more like an influx of scammers than security researchers leaning too hard on AI tools. The main problem is the bounty structure. And I don’t think these influx of low quality reports will go away, or even get any less aggressive as long as there is money to attract the scammers. Perhaps these bug bounty programs need to develop an automatic pass/fail tester of all submitted bug code, to ensure the report…

It's unfortunately widespread. We don't offer bug bounties, but we still get obviously LLM-generated "security reports" which are just nonsense and waste our time. I think the motivation may be trying to get credit for contributing to open source projects.

Re: Curl: We still have not seen a valid security report done with AI help

#236

Post by senordevnyc »

Analemma_ wrote: post #156
emushack wrote: post #49

Earlier quoted context omitted.

Reputation systems for this kind of thing sounds like rubbing some anti-itch cream on bullet wound. I feel like the problem seems to me to be behavior, not a technology issue. Personally I can't imagine how miserable it would be for my hard-earned expertise to be relegated to sifting through SLOP where maybe 1 in hundreds or even thousands of inquiries is worth any time at all. But it also doesn't seem prudent to jus…

> I feel like the problem seems to me to be behavior, not a technology issue. To be honest, this has been a grimly satisfying outcome of the AI slop debacle. For decades, the general stance of tech has been, “there is no such thing as a behavioral/social problem, we can always fix it with smarter technology”, and AI is taking that opinion and drowning it in a bathtub. You can’t fix AI slop with technology because any…

You can’t fix AI slop with technology because anything you do to detect it will be incorporated into better models until they evade your tests.

How is that a bad thing? At a certain point, it’s no longer AI slop!

https://xkcd.com/810/

Re: Curl: We still have not seen a valid security report done with AI help

#237

Post by krapp »

senordevnyc wrote: post #236
Analemma_ wrote: post #156

Earlier quoted context omitted.

> I feel like the problem seems to me to be behavior, not a technology issue. To be honest, this has been a grimly satisfying outcome of the AI slop debacle. For decades, the general stance of tech has been, “there is no such thing as a behavioral/social problem, we can always fix it with smarter technology”, and AI is taking that opinion and drowning it in a bathtub. You can’t fix AI slop with technology because any…

You can’t fix AI slop with technology because anything you do to detect it will be incorporated into better models until they evade your tests. How is that a bad thing? At a certain point, it’s no longer AI slop! https://xkcd.com/810/

Polite slop is still slop.

Most people use platforms like HN to engage in conversation with other people, not simply to assimilate information as efficiently as possible. That they are conversing with actual human beings has value to them, even when they do human things like express emotions and humor.

Hacker News could be perfectly civil if it removed the human element entirely and had an AI post links and generate threads, avoiding common tropes and boilerplate and preferring technical and factual accuracy. Make the forum read only. It would succeed in HN's goal of avoiding Eternal September and maximizing the signal to noise ratio (to the degree that it's possible with AI,) and the technical quality and information density of threads would be superior to anything HN currently hosts on average, but it would also undermine the goal of making it worth a damn to nearly anyone.

Re: Curl: We still have not seen a valid security report done with AI help

#238

Post by throwaway2037 »

spiffyk wrote: post #41
gruez wrote: post #35

Earlier quoted context omitted.

>but the three "sources" you mention are not worth much either, much like ChatGPT. I don't think I've ever seen anyone lambasted for citing stackoverflow as a source. At best, they chastised for not reading the comments, but nowhere as much pushback as for LLMs.

I have personally always been kind of against using StackOverflow as a sole source for things. It is very often a good pointer, but it's always a good idea to cross-check with primary sources. Otherwise you get all sorts of interesting surprises, like that Razer Synapse + Docker for Windows debacle. Not to mention that you are technically not allowed to just copy-paste stuff from SO.

    > Not to mention that you are technically not allowed to just copy-paste stuff from SO.
Sure you can. Over the last ten years, I have probably copied at least 100 snippets of code from StackOverflow in my corporate code base (and included a link to the original code). The stuff that was published before Generation AI Slop started is unbeatable as a source of code snippets. I am a developer for internal CRUD apps, so we don't care about licenses (except AGPL due to FUD by legal & compliance teams). Anything goes because we do not distribute our software externally.

Re: Curl: We still have not seen a valid security report done with AI help

#239

Post by kevmo314 »

tharant wrote: post #170
kevmo314 wrote: post #159

Earlier quoted context omitted.

Whenever someone has done that to me, it's clear they didn't read the ChatGPT output either and were sending it to me as some sort of "look someone else thinks you're wrong".

Again, is it possible you and the other party have (perhaps significantly) different mental models of the domain—or maybe different perspectives of the issues involved? I get that folks can be contrarian (sadly, contrariness is probably my defining trait) but it seems unlikely that someone would argue that you’re wrong by using output they didn’t read. I see impedance mismatches regularly yet folks seem often to assu…

> is it possible you and the other party have (perhaps significantly) different mental models of the domain—or maybe different perspectives of the issues involved?

Yes, however typically if that's the case they will respond with some variant of "ChatGPT mentioned xyz so I started poking in that direction, does that make sense?" There is a markedly different response when people are using ChatGPT to try to understand better and that I have no issue with.

I get what you're suggesting but I don't think people are being malicious, it's more that the discussion has gotten too deep and they're exhausted so they'd rather opt out. In some cases yes it does mean the discussion could've been simplified, but sometimes when it's a pretty deep, technical reason it's hard to avoid.

A concrete example is we had to figure out a bug in some assembly code once and we were looking at a specific instruction. I didn't believe that instruction was wrong and I pointed at the docs suggesting it lined up with what we were observing it doing. Someone responded with "I asked ChatGPT and here's what it said: ..." without even a subsequent opinion on the output of ChatGPT. In fact, reading the output it basically restated what I said, but said engineer used that as justification to rewrite the instruction to something else. And at that point I was like y'know what, I just don't care enough.

Unsurprisingly, it didn't work, and the bug never got fixed because I lost interest in continuing the discussion too.

I think what you're describing does happen in good faith, but I think people also use the wall of text that ChatGPT produces as an indirect way to say "I don't care about your opinion on this matter anymore."

Re: Curl: We still have not seen a valid security report done with AI help

#240

Post by 0x500x79 »

bogwog wrote: post #69

If I wanted to slip a vulnerability into a major open source project with a lot of eyes on it, using AI to DDOS their vulnerability reports so they're less likely to find a real report from someone who caught me seems like an obvious (and easy) step. Looking at one of the bogus reports, it doesn't even seem like a real person. Why do this if you're not trying to gain recognition?

It should be called "Denial of Attention" attack!
Post reply on HN