Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Curl: We still have not seen a valid security report done with AI help

linkedin.com

Post reply on HN
Print view View original

221–230 of 258 posts

Re: Curl: We still have not seen a valid security report done with AI help

#221

Post by imtringued »

holuponemoment wrote: post #185
VladVladikoff wrote: post #119

Earlier quoted context omitted.

This sounds more like an influx of scammers than security researchers leaning too hard on AI tools. The main problem is the bounty structure. And I don’t think these influx of low quality reports will go away, or even get any less aggressive as long as there is money to attract the scammers. Perhaps these bug bounty programs need to develop an automatic pass/fail tester of all submitted bug code, to ensure the report…

Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.

Why charge a fee? All you need is a reputation system where low reputation bounty hunters need a reputable person to vouch for them. If it turns out to be false, both take a hit. If true, the voucher gets to be a co-author and a share in the bounty.

Re: Curl: We still have not seen a valid security report done with AI help

#222

Post by tharant »

vuserfcase wrote: post #214
tharant wrote: post #199

Earlier quoted context omitted.

In my experience, and for use-cases that are carefully considered, language models are not confidently wrong a majority of the time. The trick is understanding the tool and using it appropriately—thus the “carefully considered” approach to identifying use-cases that can provide value.

A use-case that can be carefully considered requires more knowledge about the use-case than the LLM, it requires you to understand the specific model's training and happy paths, it requires more time to make it output the thing you want than just doing it yourself. If you don't know enough about the subject or the model, you will get confident garbage

> A use-case that can be carefully considered requires more knowledge about the use-case than the LLM

I would tend to agree with that assertion…

> it requires you to understand the specific model's training and happy paths

But I strongly disagree with that assertion; I know nothing of commercial models’ training corpus, methodology, or even their system prompts; I only know how to use them as a tool for various use-cases.

> it requires more time to make it output the thing you want than just doing it yourself.

And I strongly disagree with that one too. As long as the thing you want it to output is rooted in relatively mainstream or well-known concepts, it’s objectively much faster than you/we are; maybe it’s more expensive but it’s also crazy fast—which is the point of all tools—and the precision/accuracy of most speedy tools can be often deferred until a later step in the process.

> If you don't know enough about the subject or the model, you will get confident garbage

Once you step outside their comfort zone (their training), well, yah… they do all tend to be unduly confident in their responses—I’d argue however that it is a trait they learned from us; we really like to be confident even when we’re wrong and that trait is borne out dramatically across the internet sources on which a lot of these models were trained.

Re: Curl: We still have not seen a valid security report done with AI help

#224

Post by namaria »

silversmith wrote: post #120
cogman10 wrote: post #60

Earlier quoted context omitted.

I recently had this happen from a senior engineer. What's really frustrating is I TOLD them the issues and how to fix it. Instead of listening to what I told them, they plugged it into GPT and responded with "Oh, interesting this is what GPT says" (Which, spoiler, was similar but lacking from what I'd said). Meaning, instead of listening to a real-life expert in the company telling them how to handle the problem they…

I often do this - ask a LLM for an answer when I already have it from an expert. I do it to evaluate the ability of the LLM. Usually not in the presence of said expert tho.

Just using LLMs on the (few) things I have specialist knowledge of it's clear they are extremely limited. I get absurdly basic mistakes and I am very wary of even reading LLM output about topics I don't command. It's easy to get stuck on dead ends reasoning wise even by getting noisy input.

Re: Curl: We still have not seen a valid security report done with AI help

#225

Post by namaria »

tharant wrote: post #199
parliament32 wrote: post #195

Earlier quoted context omitted.

Because calculators aren't confidently wrong the majority of the time.

In my experience, and for use-cases that are carefully considered, language models are not confidently wrong a majority of the time. The trick is understanding the tool and using it appropriately—thus the “carefully considered” approach to identifying use-cases that can provide value.

In the very narrow fields where I have a deep understanding, LLM output is mostly garbage. It sounds plausible but doesn't stand up to scrutiny. The basics that it can regurgitate from wikipedia sound mostly fine but they are already subtly wrong as soon as they depart from stating very basic facts.

Thus I have to assume that for any topic I do not fully understand - which is the vast majority of human knowledge - it is worse than useless, it is actively misleading. I try to not even read much of what LLMs produce. I might give it some text and riff about it if I need ideas, but LLMs are categorically the wrong tool for factual content.

Re: Curl: We still have not seen a valid security report done with AI help

#227

Post by [deleted] »

pixl97 wrote: post #93
colecut wrote: post #76

Earlier quoted context omitted.

That rule does not imply the inverse

I mean we have automated systems that 'write' things like tornado warnings. Would you rather we have someone hand write that out? It seems the initial rule seems rather worthless.

[deleted]

Re: Curl: We still have not seen a valid security report done with AI help

#228

Post by namaria »

alwa wrote: post #92
yoyohello13 wrote: post #25

Earlier quoted context omitted.

Seriously. Being able to look up stuff using AI is not unique. I can do that too. This is kind of the same with any AI gen art. Like I can go generate a bunch of cool images with AI too, why should I give a shit about your random Midjourney output.

I mean… I have a fancy phone camera in my pocket too, but there are photographers who, with the same model of fancy phone camera, do things that awe and move me. It took a solid hundred years to legitimate photography as an artistic medium, right? To the extent that the controversy still isn’t entirely dead? Any cool images I ask AI for are going to involve a lot less patience and refinement than some of these things…

> It took a solid hundred years to legitimate photography as an artistic medium, right?

Not really.

"In 1853 the Photographic Society, parent of the present Royal Photographic Society, was formed in London, and in the following year the Société Française de Photographie was founded in Paris."

https://www.britannica.com/technology/photography/Photograph...

Re: Curl: We still have not seen a valid security report done with AI help

#229

Post by cedws »

holuponemoment wrote: post #185
VladVladikoff wrote: post #119

Earlier quoted context omitted.

This sounds more like an influx of scammers than security researchers leaning too hard on AI tools. The main problem is the bounty structure. And I don’t think these influx of low quality reports will go away, or even get any less aggressive as long as there is money to attract the scammers. Perhaps these bug bounty programs need to develop an automatic pass/fail tester of all submitted bug code, to ensure the report…

Simply charge a fee to submit a report. At 1% of the payment for low bounties it's perfectly valid. Maybe progressively scale that down a bit as the bounty goes up. But still for a $50k bounty you know is correct it's only $500.

If you charge a fee the motivation for good samaritan reports goes to zero.

Re: Curl: We still have not seen a valid security report done with AI help

#230

Post by ndsipa_pomu »

seanp2k2 wrote: post #211
3abiton wrote: post #205

Earlier quoted context omitted.

This is a whole new problem open source project will be facing. AI slop PR and Vulnerability reports, which will be only solved using AI tools to filter through the unholy amount.

AI filtering AI that's submitted based on AI scouring the web for ways to make probably less money than it costs to run. The future looks like turning on computers and having them run at 100% GPU + CPU usage 100% of time with 0 clue what they're doing. What a future.

Sounds a bit like what happens in Colossus: The Forbin Project https://www.imdb.com/title/tt0064177/
Post reply on HN