Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability reports are not special anymore

words.filippo.io

Post reply on HN
Print view View original

191–200 of 249 posts

Re: Vulnerability reports are not special anymore

#191

Post by pwinnski »

socalgal2 wrote: post #34

I feel like the current situation is temporary. LLMs are finding all the bugs. LLMs are also help fixing most of the bugs. Once most of the bugs are fixed, LLMs should be good at finding bugs before shipping them, the stream of bug reports will die down, and we'll be back to vulnerabiltiy reports being special. Further, the fact that bugs are so easy to find by LLMs means there is strong incentives to find ways to mi…

It may be only a matter of time before all devs remember to append ", and make no mistakes" to the end of their LLM prompts, but I don't think we as an industry will ever reach a point at which every release of every package/library/application is scanned with the most capable model available.

I mean, we've had tooling like fuzzers available for a very long time, and most devs haven't run one against their software ever, let alone before each release.

It's the human factor I think will keep this a problem essentially-forever.

Re: Vulnerability reports are not special anymore

#192

Post by TeMPOraL »

Quothling wrote: post #79
jamesfinlayson wrote: post #51

Earlier quoted context omitted.

I tried to raise that with my internal security team recently - don't clutter my vulnerability dashboard with issues in dev dependencies. They somewhat rightly pointed out that malware needs to be dealt even if it's a dev dependency. So my suggestion went nowhere because I guess we can't filter by type of vulnerability.

Working in the EU energy sector where we have to work with NIS2 compliance, I'd argue that your security team rightly pointed it out. I suspect that's what you mean though, and the rightly is just there because you agree with it but don't like it. We work with even more tight dependencies policies than just having alerts. We have a set of pre-approved and yearly vetted packages, like pandas or pyarrow for Python data…

> compliance basically comes down to "We trust the [third party] enough, and [third party component] is vital to our business, so we accept the risks"

This is the beginning and end of reasonable security. This is what it's always about, and if you go beyond it, you risk practicing art for the sake of art, at the expense of customers and other stakeholders.

Security is about understanding and managing risk. Not about achieving some mathematical perfection (which is actually only achievable by making your system an inert piece of rock, but people realize that way too late, after piling on way too many pointless "security improvements").

Re: Vulnerability reports are not special anymore

#193

Post by [deleted] »

swiftcoder wrote: post #127
themanmaran wrote: post #2

I feel like it's also been overrun by a lot of spam. As someone running a company, I get 2-5 unsolicited "vulnerability reports" per week. Half of them are an LLM finding some bad CSS on our framer splash page. The other half I assume are an extortion attempt so we just mark as spam. Occasionally I see real security researchers on HN complaining that no one takes the disclosure seriously, or that people reply immedia…

We also get unsolicited vulnerability reports from companies trying to poach our annual pentest contract, which is... a tad grey ethically-speaking

[deleted]

Re: Vulnerability reports are not special anymore

#194

Post by [deleted] »

bostik wrote: post #111
socalgal2 wrote: post #34

I feel like the current situation is temporary. LLMs are finding all the bugs. LLMs are also help fixing most of the bugs. Once most of the bugs are fixed, LLMs should be good at finding bugs before shipping them, the stream of bug reports will die down, and we'll be back to vulnerabiltiy reports being special. Further, the fact that bugs are so easy to find by LLMs means there is strong incentives to find ways to mi…

"Temporary" can be an awfully long time. There is ample evidence that discovery rate of bugs (many of which can be bucketed into vulnerabilities) in any non-trivial piece of software is more or less stable.[0] In a recent podcast episode the ex-CISO of Adobe commented that every now and then they'd take a sustained squeeze to find all occurrences of a given type of bug (ie. source of vulnerability) in a codebase. The…

[deleted]

Re: Vulnerability reports are not special anymore

#195

Post by throwaway2037 »

entuno wrote: post #187
throwaway2037 wrote: post #163

Earlier quoted context omitted.

> chance of being arrested I am not involved with security research in any way. Can you explain the threat here?

There is a history of companies and organisations threatening legal action against security researchers when they report vulnerabilities in their systems or products. Sometimes even when the testing has been completely offline - I know people who have downloaded some software, carried out testing against a local copy of it, and then faced legal threats when they tried to report serious security vulnerabilities to the…

Yikes. Thanks for the good faith reply. Does EFF help to defend some of these cases?

Re: Vulnerability reports are not special anymore

#196

Post by jcgrillo »

shakna wrote: post #68
jcgrillo wrote: post #55

Earlier quoted context omitted.

The problem always existed, but nobody amassed a sufficiently large army of trolls to exploit it until now. So it wasn't a priority to solve it before, but now it is. We're going to have to learn to differentiate reports that matter from those that don't. Classifying reports might actually be something you could productively use an LLM for..

When we solved this problem for email... We just dumped everything similar to untrusted into the rubbish bin. Important things can vanish too, and that's an acceptable price.

Yeah unfortunately there will be some high quality reports that get erroneously discarded, but so long as the signal/noise ratio improves sufficiently that's an acceptable price.

Re: Vulnerability reports are not special anymore

#197

Post by iso1631 »

adamddev1 wrote: post #185
ttoinou wrote: post #173

Earlier quoted context omitted.

Okay, so the whole world is failing and success doesn't exist. Maybe now you need a better criteria than your previous failure / success_100%_infiniteCost to observe and understand the the world

This logic enables and encourages corruption in all spheres of life. "Things are messed up as they are, so why try so hard to do the right thing?" "Let's be realistic."

The logic allows that to continue because "good" and "better" isn't good enough, only "perfect" is allowed, so instead "nothing" is done

Re: Vulnerability reports are not special anymore

#198

Post by inigyou »

throwaway2037 wrote: post #163
inigyou wrote: post #53

Earlier quoted context omitted.

Why would anyone pay money to have a chance of being arrested?

> chance of being arrested I am not involved with security research in any way. Can you explain the threat here?

When you report a vulnerability in a product that means you hacked the product. Hacking is illegal. If it's something that runs on your own computer you might get away but if it runs on a server then it's 100% a felony.

Re: Vulnerability reports are not special anymore

#199

Post by inigyou »

ozim wrote: post #175
inigyou wrote: post #142

Earlier quoted context omitted.

Why not?

Because you are not providing any service not selling anything. There is no real way as a company to withhold someone’s money and that it goes through accounting. I am not an accountant so ask some accountants why not.

it's not illegal to ask people to send you money and then keep the money they send you

Re: Vulnerability reports are not special anymore

#200

Post by gautamsomani »

TeMPOraL wrote: post #192
Quothling wrote: post #79

Earlier quoted context omitted.

Working in the EU energy sector where we have to work with NIS2 compliance, I'd argue that your security team rightly pointed it out. I suspect that's what you mean though, and the rightly is just there because you agree with it but don't like it. We work with even more tight dependencies policies than just having alerts. We have a set of pre-approved and yearly vetted packages, like pandas or pyarrow for Python data…

> compliance basically comes down to "We trust the [third party] enough, and [third party component] is vital to our business, so we accept the risks" This is the beginning and end of reasonable security. This is what it's always about, and if you go beyond it, you risk practicing art for the sake of art, at the expense of customers and other stakeholders. Security is about understanding and managing risk. Not about…

Asking out of curiousity - how would you or how does your org handles this right now?
Post reply on HN