Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl: We still have not seen a valid security report done with AI help

linkedin.com

Post reply on HN
Print view View original

191–200 of 258 posts

Re: Curl: We still have not seen a valid security report done with AI help

#191

Post by mwigdahl »

jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

I do this, but it’s because I am evangelizing proper use of the tool to developers who don’t always understand what it can and can’t do.

Recently I used o3 to plan a refactoring related to upgrading the version of C++ we are using in our product. It pointed out that we could use a tool built in to VS 2022 to make a particular change automatically based on compilation output. I was not familiar with this tool and neither were the other developers on the team.

I did confirm its accuracy myself, but also made sure to credit the model as the source of information about the tool.

Re: Curl: We still have not seen a valid security report done with AI help

#192

Post by bigiain »

soraminazuki wrote: post #94
parliament32 wrote: post #22

Didn't even have to click through to the report in question to know it would be all hallucinations -- both the original patchfile and the segfault ("ngtcp2_http3_handle_priority_frame".. "There is no function named like this in current ngtcp2 or nghttp3.") I guess these guys don't bother to verify, they just blast out AI slop and hope one of them hits?

Considering that even the reporter responded to requests for clarification with yet another AI slop, they likely lack the technical background.

"they likely lack the ethical background."

FTFY

Re: Curl: We still have not seen a valid security report done with AI help

#193

Post by tharant »

stevage wrote: post #179
tharant wrote: post #174

Earlier quoted context omitted.

> I really fear that a number of engineers are going to us GPT to avoid thinking. They view it as a shortcut to problem solve and it isn't. How is this sentiment not different from my grandfather’s sentiment that calculators and computers (and probably his grandfather’s view of industrialization) are a shortcut to avoid work? From my perspective most tools are used as a shortcut to avoid work; that’s kinda the while…

Did you grandpa think that calculators made engineers worse at their jobs?

I don’t know for certain (he’s no longer around) but I suspect he did. The prevalence of folks who nowadays believe that Gen-AI makes everything worse suggests to me that not much has changed since his time.

I get it; I’m not an AI evangelist and I get frustrated with the slop too; Gen-AI (and many of the tools we’ve enjoyed over the past few millennia) was/is lauded as “The” singular tool that makes everything better; no tool can fulfill that role yet we always try to shoehorn our problems into a shape that fits the tool. We just need to use the correct tools for the job; in my mind, the only problem right now is that we have a really capable tool and have identified some really valuable use-cases for that tool yet we also keep trying to use it for (what I believe are, given current capabilities) use-cases that don’t fit the tool.

We’ll figure it out but, in the meantime, while I don’t like to generalize that a tech or its use-cases are objectively good/bad, I do tend to have an optimistic outlook for most tech—Gen-AI included.

Re: Curl: We still have not seen a valid security report done with AI help

#194

Post by bigiain »

gorbachev wrote: post #99
rdtsc wrote: post #65

> evilginx updated the severity from none to high Well the reporter in the report that stated it that they are open for employment https://hackerone.com/reports/3125832 Anyone want to hire them? They can play with ChatGPT all day and spam random projects with the AI slop.

Growth hack: hire this person to find vulnerabilities in competitors' products.

Effective altruism: hire this guy to manipulate software company's stock prices with highly publicized "vulnerabilities" in their products...

Re: Curl: We still have not seen a valid security report done with AI help

#195

Post by parliament32 »

tharant wrote: post #174
cogman10 wrote: post #60

Earlier quoted context omitted.

I recently had this happen from a senior engineer. What's really frustrating is I TOLD them the issues and how to fix it. Instead of listening to what I told them, they plugged it into GPT and responded with "Oh, interesting this is what GPT says" (Which, spoiler, was similar but lacking from what I'd said). Meaning, instead of listening to a real-life expert in the company telling them how to handle the problem they…

> I really fear that a number of engineers are going to us GPT to avoid thinking. They view it as a shortcut to problem solve and it isn't. How is this sentiment not different from my grandfather’s sentiment that calculators and computers (and probably his grandfather’s view of industrialization) are a shortcut to avoid work? From my perspective most tools are used as a shortcut to avoid work; that’s kinda the while…

Because calculators aren't confidently wrong the majority of the time.

Re: Curl: We still have not seen a valid security report done with AI help

#196

Post by jasinjames »

unsnap_biceps wrote: post #16

For those of you who don't want to click into linked in, https://hackerone.com/reports/3125832 is the latest example of a invalid curl report

Can someone explain the ip address in the hackerone profile[0]? I can't tell if 139.224.130.174 is a reference to something real or just hallucinated by the LLM to look "cool". Wikipedia says that this /8 is controlled by "MIX"[1] but my google-fu is failing me atm.

[0] https://hackerone.com/evilginx?type=user [1] https://en.wikipedia.org/wiki/List_of_assigned_/8_IPv4_addre...

Re: Curl: We still have not seen a valid security report done with AI help

#197

Post by parliament32 »

Frost1x wrote: post #129
jacksnipe wrote: post #15

Something that really frustrates me about interacting with (some) people who use AI a lot is that they will often tell me things that start “I asked ChatGPT and it said…” stop it!!! If the chatbot taught you something and you understood it, explain it to me. If you didn’t understand or didn’t trust it, then keep it to yourself!

I work in a corporate environment as I’m sure many others do. Many executives have it in their head that LLMs are this brand new efficiency gain they can pad profit margins with, so you should be using it for efficiency. There’s a lot of push for that, everywhere where I work. I see email blasts suggesting I should be using it, I get peers saying I should be using it, I get management suggesting I should use it to cu…

> I see email blasts suggesting I should be using it, I get peers saying I should be using it, I get management suggesting I should use it to cut costs

The ideal scenario: you write a few bulletpoints and ask Copilot to turn it into a long-form email to send out. Your receiving coworker then asks Copliot to distill it back into a few bullet points they can skim.

You saved 5 minutes but one of your points was ignored entirely and 20% of your output is nonsensical.

Your coworker saved 2 minutes but one of their bulletpoints was hallucinated and important context is missing from the others.

Microsoft collects a fee from both of you and is the only winner here.

Re: Curl: We still have not seen a valid security report done with AI help

#198

Post by cogman10 »

tharant wrote: post #149
cogman10 wrote: post #60

Earlier quoted context omitted.

I recently had this happen from a senior engineer. What's really frustrating is I TOLD them the issues and how to fix it. Instead of listening to what I told them, they plugged it into GPT and responded with "Oh, interesting this is what GPT says" (Which, spoiler, was similar but lacking from what I'd said). Meaning, instead of listening to a real-life expert in the company telling them how to handle the problem they…

Is it possible that what happened was an impedance mismatch between you and the engineer such that they couldn’t grok what you told them but ChatGPT was able to describe it in a manner they could understand? Real-life experts (myself included, though I don’t claim to be an expert in much) sometimes have difficulty explaining domain-specific concepts to other folks; it’s not a flaw in anyone, folks just have different…

Definitely a possibility.

However, I have a very strong suspicion they also didn't understand the GPT output.

To flush out the situation a bit further, this was a performance tuning problem with highly concurrent code. This engineer was initially tasked with the problem and they hadn't bothered to even run a profiler on the code. I did, shared my results with them, and the first action they took with my shared data was dumping a thread dump into GPT and asking it where the performance issues were.

Instead, they've simply been littering the code with timing logs in hopes that one of them will tell them what to do.

Re: Curl: We still have not seen a valid security report done with AI help

#199

Post by tharant »

parliament32 wrote: post #195
tharant wrote: post #174

Earlier quoted context omitted.

> I really fear that a number of engineers are going to us GPT to avoid thinking. They view it as a shortcut to problem solve and it isn't. How is this sentiment not different from my grandfather’s sentiment that calculators and computers (and probably his grandfather’s view of industrialization) are a shortcut to avoid work? From my perspective most tools are used as a shortcut to avoid work; that’s kinda the while…

Because calculators aren't confidently wrong the majority of the time.

In my experience, and for use-cases that are carefully considered, language models are not confidently wrong a majority of the time. The trick is understanding the tool and using it appropriately—thus the “carefully considered” approach to identifying use-cases that can provide value.

Re: Curl: We still have not seen a valid security report done with AI help

#200

Post by justsid »

holuponemoment wrote: post #185
VladVladikoff wrote: post #119

Earlier quoted context omitted.

This sounds more like an influx of scammers than security researchers leaning too hard on AI tools. The main problem is the bounty structure. And I don’t think these influx of low quality reports will go away, or even get any less aggressive as long as there is money to attract the scammers. Perhaps these bug bounty programs need to develop an automatic pass/fail tester of all submitted bug code, to ensure the report…

Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.

Could also be made refundable when the bug report is found to be valid. Although of course the problem then becomes some kid somewhere who is into computers and hacking find something but can’t easily report it because the barrier to entry is too high now. I don’t think there is a good solution unfortunately.
Post reply on HN