Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability reports are not special anymore

words.filippo.io

Post reply on HN
Print view View original

81–90 of 249 posts

Re: Vulnerability reports are not special anymore

#81

Post by zemblanKing »

socalgal2 wrote: post #34

I feel like the current situation is temporary. LLMs are finding all the bugs. LLMs are also help fixing most of the bugs. Once most of the bugs are fixed, LLMs should be good at finding bugs before shipping them, the stream of bug reports will die down, and we'll be back to vulnerabiltiy reports being special. Further, the fact that bugs are so easy to find by LLMs means there is strong incentives to find ways to mi…

I feel this sentiment is wishful thinking,but I want to start by saying I hope it turns out to be correct.

I find that often bugs will be created when using an LLM, like others have said. Saying that this can then be fixed by identifying all the bugs created by an LLM with an LLM doesnt guarantee another bug is not introduced when the LLM is addressing the initial problem.

Also, what if the LLM has a blind spot. They certainly also could be incapable of finding or fixing a bug. They dont pass any benchmark at 100% right now. Also also, guaranteeing there are no bugs in your code is like saying you have 100% test coverage, all of the tests pass, and they are written perfectly. Saying that you can simply identify and fix the bugs also assumes there is enough time and energy to find all of the bugs that exist within a project and then to address them. Even LLMs use time and energy. In a sufficiently complex system that is certainly wishful thinking.

Considering the size and complexity of a lot of modern software (like web browsers, 3d modelling software, game engines, etc.) software is just too complex to not have bugs even when created and managed by LLMs.

There will continue to be bugs in code and we will simply have to live with the fact that LLMs make it easier to exploit computer systems. I mean consider a hardware bug like Spectre [0]. If bugs like this become easier to find does that mean our existing hardware will just become obsolete more quickly? that type of problem can be addressed, but at quite a high cost.

Not sure what all of this means for the future.

0. https://en.wikipedia.org/wiki/Spectre_%28security_vulnerabil...

Re: Vulnerability reports are not special anymore

#82

Post by moi2388 »

fragmede wrote: post #28
Avicebron wrote: post #20

Earlier quoted context omitted.

> Chess players manage to do it at least. The 5 on earth still getting paid to play chess?

There's only one Magnus Carlsen, who earned > $1 million in 2025 for playing chess, but the long tail, there were 26 people who made more than $100k, https://thechessworld.com/articles/general-information/the-1... but like, if you mean literally "someone gave them money and they played a game of chess", the number becomes much bigger. Chess coaches, streamers, club instructors, exhibition players, league players, cam…

And if you mean “people who can live off of tournament winnings” it’s not more than 26.

It’s like most of art, writing, and sports. The only way to make money is by becoming a teacher.

Re: Vulnerability reports are not special anymore

#83

Post by fouc »

inigyou wrote: post #53
lemagedurage wrote: post #50

Earlier quoted context omitted.

Have you considered requiring a small payment for vulnerability disclosure? Refund it on payout. This should be very effective at deterring spammers. It also sucks for real reports, but beats shutting down the program entirely.

Why would anyone pay money to have a chance of being arrested?

why would anyone accept bounty money to have a chance of being arrested?

Re: Vulnerability reports are not special anymore

#85

Post by gorgoiler »

jamesfinlayson wrote: post #54
Gigachad wrote: post #6

Earlier quoted context omitted.

I'm getting CVE fatigue with all of these super ultra critical 10/10 vulnerabilities that are some node package that compiles my frontend can get stuck if I give it a malicious regex. It's hard to spot the stuff that actually matters.

Yep. I remember years ago seeing the website for some guy who proudly listed all the CVEs he'd discovered. Clearly he'd written some scanning tool to look at regexes in open source projects and was creating CVEs for anything that might result in exponential time execution or whatever.

It sounds like an interesting case study. Do these things get reported with a patch?:

(a) add a new function that does regular expressions searching / matching with a resource checker (eg a timer);

(b) write a local linter that reports an error for any use of the builtin regular expression tools;

(c) fix all the lint warnings;

(d) commit the linter.

Re: Vulnerability reports are not special anymore

#86

Post by jamesfinlayson »

tempay wrote: post #80
zmgsabst wrote: post #67

Earlier quoted context omitted.

Dev dependencies is how they compromised SolarWinds and thereby most of the US federal government. > The attackers used a supply chain attack. The attackers accessed the build system belonging to the software company SolarWinds, possibly via SolarWinds's Microsoft Office 365 account, which had also been compromised at some point. SolarWinds was using build management and continuous integration server TeamCity provide…

This is very real, but such CVEs are such a tiny fraction in relation to denial-of-service-due-to-regex that it’s hard to take the system seriously. So far as I’m concerned the solution is to isolate everything as much as possible. I’d love to see something on the CVE classification side to also address the signal to noise problem but I don’t see it happening.

Pretty much - I don't know too much about the CVE process but if ReDoS stuff was flagged at the CVE level as "exploitable only with unconstrained inputs" then great - I know my tests have sane inputs, so I'll close thanks.

Re: Vulnerability reports are not special anymore

#87

Post by jamesfinlayson »

Quothling wrote: post #79
jamesfinlayson wrote: post #51

Earlier quoted context omitted.

I tried to raise that with my internal security team recently - don't clutter my vulnerability dashboard with issues in dev dependencies. They somewhat rightly pointed out that malware needs to be dealt even if it's a dev dependency. So my suggestion went nowhere because I guess we can't filter by type of vulnerability.

Working in the EU energy sector where we have to work with NIS2 compliance, I'd argue that your security team rightly pointed it out. I suspect that's what you mean though, and the rightly is just there because you agree with it but don't like it. We work with even more tight dependencies policies than just having alerts. We have a set of pre-approved and yearly vetted packages, like pandas or pyarrow for Python data…

Yeah I completely understand their intent, but I might get 30 vulnerabilities across a multiple repos flagged in a week. It is already tedious to check them all and assess if they're worth worrying about let alone having to update them. These are 99% Javascript though - I suspect other ecosystems are much more manageable.

Re: Vulnerability reports are not special anymore

#88

Post by jamesfinlayson »

gorgoiler wrote: post #85
jamesfinlayson wrote: post #54

Earlier quoted context omitted.

Yep. I remember years ago seeing the website for some guy who proudly listed all the CVEs he'd discovered. Clearly he'd written some scanning tool to look at regexes in open source projects and was creating CVEs for anything that might result in exponential time execution or whatever.

It sounds like an interesting case study. Do these things get reported with a patch?: (a) add a new function that does regular expressions searching / matching with a resource checker (eg a timer); (b) write a local linter that reports an error for any use of the builtin regular expression tools; (c) fix all the lint warnings; (d) commit the linter.

No I think he was just looking to raise his profile, not to help.

Re: Vulnerability reports are not special anymore

#90

Post by Gigachad »

jamesfinlayson wrote: post #87
Quothling wrote: post #79

Earlier quoted context omitted.

Working in the EU energy sector where we have to work with NIS2 compliance, I'd argue that your security team rightly pointed it out. I suspect that's what you mean though, and the rightly is just there because you agree with it but don't like it. We work with even more tight dependencies policies than just having alerts. We have a set of pre-approved and yearly vetted packages, like pandas or pyarrow for Python data…

Yeah I completely understand their intent, but I might get 30 vulnerabilities across a multiple repos flagged in a week. It is already tedious to check them all and assess if they're worth worrying about let alone having to update them. These are 99% Javascript though - I suspect other ecosystems are much more manageable.

It's easier to keep stuff up to date these days. If you have a project with typescript, unit tests, and end to end tests like cypress you can just have dependabot create the PRs to update packages. If everything passes you just have to hit the merge button.

Just updating everything is probably easier than assessing if it's possible to trigger an exploit with the way you use the package.

Post reply on HN