Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl: We still have not seen a valid security report done with AI help

linkedin.com

Post reply on HN
Print view View original

211–220 of 258 posts

Re: Curl: We still have not seen a valid security report done with AI help

#211

Post by seanp2k2 »

3abiton wrote: post #205
terom wrote: post #168

Earlier quoted context omitted.

The git commit hashes in the diff are interesting: 1a2b3c4..d4e5f6a I think my wetware pattern-matching brain spots a pattern there.

This is a whole new problem open source project will be facing. AI slop PR and Vulnerability reports, which will be only solved using AI tools to filter through the unholy amount.

AI filtering AI that's submitted based on AI scouring the web for ways to make probably less money than it costs to run. The future looks like turning on computers and having them run at 100% GPU + CPU usage 100% of time with 0 clue what they're doing. What a future.

Re: Curl: We still have not seen a valid security report done with AI help

#212

Post by seanp2k2 »

xk_id wrote: post #206
unsnap_biceps wrote: post #16

For those of you who don't want to click into linked in, https://hackerone.com/reports/3125832 is the latest example of a invalid curl report

Not sure what timeline this is anymore where a tech website loads up a completely blank page on my mobile device.

Welcome to the web in 2025, where it takes 5MB of JS and everything else to load a blog post containing 640B of text.

Re: Curl: We still have not seen a valid security report done with AI help

#213

Post by phs318u »

danielvf wrote: post #101

I handle reports for a one million dollar bug bounty program. AI spam is bad. We've also never had a valid report from an by an LLM (that we could tell). People using them will take any being told why a bug report is not valid, questions, or asks for clarification and run them back through the same confused LLM. The second pass through generates even deeper nonsense. It's making even responding with anything but "clo…

>It's the people that concern me. They cannot tell the difference between truth and garbage.

Suffice to say, this statement is an accurate assessment of the current state of many more domains than merely software security.

Re: Curl: We still have not seen a valid security report done with AI help

#214

Post by vuserfcase »

tharant wrote: post #199
parliament32 wrote: post #195

Earlier quoted context omitted.

Because calculators aren't confidently wrong the majority of the time.

In my experience, and for use-cases that are carefully considered, language models are not confidently wrong a majority of the time. The trick is understanding the tool and using it appropriately—thus the “carefully considered” approach to identifying use-cases that can provide value.

A use-case that can be carefully considered requires more knowledge about the use-case than the LLM, it requires you to understand the specific model's training and happy paths, it requires more time to make it output the thing you want than just doing it yourself. If you don't know enough about the subject or the model, you will get confident garbage

Re: Curl: We still have not seen a valid security report done with AI help

#215

Post by vuserfcase »

colechristensen wrote: post #87
cogman10 wrote: post #60

Earlier quoted context omitted.

I recently had this happen from a senior engineer. What's really frustrating is I TOLD them the issues and how to fix it. Instead of listening to what I told them, they plugged it into GPT and responded with "Oh, interesting this is what GPT says" (Which, spoiler, was similar but lacking from what I'd said). Meaning, instead of listening to a real-life expert in the company telling them how to handle the problem they…

>They view it as a shortcut to problem solve and it isn't Oh but it is, used wisely. One: it's a replacement for googling a problem and much faster. Instead of spending half an hour or half a day digging through bug reports, forum posts, and stack overflow for the solution to a problem. LLMs are a lot faster, occasionally correct, and very often at least rather close. Two: it's a replacement for learning how to do so…

>Oh but it is, used wisely.

Sufficiently advanced orange juice extractor is the solution to any problem. Doesen't necessarily mean you should build the sufficient part.

>One: it's a replacement for googling a problem and much faster

This is more to do with the problem that google results have gone downhill very rapidly. It used to be you could find what you were looking for very fast and solve a problem.

>I could ask an LLM to do it and then tweak the results. Five minutes for "good enough" and it really is.

When the cost of failures is low, a hackjob can be economical, like a generated picture for entertainment or a static error page. Miscreating a support for a bridge it is not very economical

Re: Curl: We still have not seen a valid security report done with AI help

#216

Post by darkoob12 »

unsnap_biceps wrote: post #16

For those of you who don't want to click into linked in, https://hackerone.com/reports/3125832 is the latest example of a invalid curl report

You can tell it's ChatGPT from the stupid icon. In one of the iterations they started using thses emojis which are disturbing for me. The answer to the first question has obvious ChatGPT writing style.

Re: Curl: We still have not seen a valid security report done with AI help

#217

Post by ponector »

holuponemoment wrote: post #185
VladVladikoff wrote: post #119

Earlier quoted context omitted.

This sounds more like an influx of scammers than security researchers leaning too hard on AI tools. The main problem is the bounty structure. And I don’t think these influx of low quality reports will go away, or even get any less aggressive as long as there is money to attract the scammers. Perhaps these bug bounty programs need to develop an automatic pass/fail tester of all submitted bug code, to ensure the report…

Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.

You are adding more incentive to go directly to black market to sell vulnerability.

Also I've heard many times cases when company refused to pay bounty for any reason.

And taxes, how you'll tax it internationally? Sales tax? VAT?

Re: Curl: We still have not seen a valid security report done with AI help

#218

Post by Seb-C »

danielvf wrote: post #101

I handle reports for a one million dollar bug bounty program. AI spam is bad. We've also never had a valid report from an by an LLM (that we could tell). People using them will take any being told why a bug report is not valid, questions, or asks for clarification and run them back through the same confused LLM. The second pass through generates even deeper nonsense. It's making even responding with anything but "clo…

> I believe that one day there will be great code examining security tools.

As for programming, I think that we will simply continue to have incrementally better tools based on sane and appropriate technologies, as we have had forever.

What I'm sure about is that no such tool can come out of anything based on natural language, because it's simply the worst possible interface to interact with a computer.

Re: Curl: We still have not seen a valid security report done with AI help

#219

Post by Seb-C »

vessenes wrote: post #34

Reading the straw that broke the camel's back commit illustrates the problem really well: https://hackerone.com/reports/3125832 . This shit must be infuriating to dig through. I wonder if reputation systems might work here - you could give anyone who id's with an AML/KYC provider some reputation, enough for two or three reports, let people earn reputation digging through zero rep submissions and give someone like 10,…

IMO, this AI crap is just the next step of the "let's block criminal behavior with engineering" path we followed for decades. That might very well be the last straw, as it is very unlikely we can block this one efficiently and reliably.

It's due time we ramp-up our justice systems to make people truly responsible and punished for their bad behavior online, including all kind of spams, scams, fishing and disinformation.

That might involve the end of anonymity on internet, and lately I feel that the downsides of that are getting smaller and smaller compared to it's upsides.

Re: Curl: We still have not seen a valid security report done with AI help

#220

Post by 59nadir »

cogman10 wrote: post #198
tharant wrote: post #149

Earlier quoted context omitted.

Is it possible that what happened was an impedance mismatch between you and the engineer such that they couldn’t grok what you told them but ChatGPT was able to describe it in a manner they could understand? Real-life experts (myself included, though I don’t claim to be an expert in much) sometimes have difficulty explaining domain-specific concepts to other folks; it’s not a flaw in anyone, folks just have different…

Definitely a possibility. However, I have a very strong suspicion they also didn't understand the GPT output. To flush out the situation a bit further, this was a performance tuning problem with highly concurrent code. This engineer was initially tasked with the problem and they hadn't bothered to even run a profiler on the code. I did, shared my results with them, and the first action they took with my shared data w…

I'm sorry, how is this a "senior engineer"? Is this a "they worked in the industry for 6 years and are now senior" type situation or are they an actual senior engineer? Because it seems like they're lacking the basics to work on what you yourself seem to consider senior engineer problems for your project.

Also, what is your history and position in the company? It seems odd that you'd get completely ignored by this supposed senior engineer (something that usually happens more often with overconfident juniors) if you have meaningful experience in the field and domain.

Post reply on HN