Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability reports are not special anymore

words.filippo.io

Post reply on HN
Print view View original

141–150 of 249 posts

Re: Vulnerability reports are not special anymore

#141

Post by wongarsu »

socalgal2 wrote: post #92
zemblanKing wrote: post #81

Earlier quoted context omitted.

I feel this sentiment is wishful thinking,but I want to start by saying I hope it turns out to be correct. I find that often bugs will be created when using an LLM, like others have said. Saying that this can then be fixed by identifying all the bugs created by an LLM with an LLM doesnt guarantee another bug is not introduced when the LLM is addressing the initial problem. Also, what if the LLM has a blind spot. They…

If LLMs can trivially find bugs, then they can trivially find bugs. If they can't find any bugs that doesn't mean there are no bugs but it suggests that others can't easily find them either. So the LLMs find all the bugs problem is fixed by asking the LLMs to find them before you ship. Read what wrote, I didn't say your program will be bug free. I said, if the LLM can trivally find the bug it will. If it can't then w…

The bugs I can trivially find, the bugs Claude can trivially find and the bugs Codex can trivially find are not necessarily the same. The most obvious bugs would be obvious to any of us three, but beyond that we would't agree on a definition of 'trivial bug'.

This gets worse if you factor in different harnesses built for the task, and future model updates

Sure, things will get calmer than they are now, but shipping without bugs someone else considers trivial will still require more effort than most are willing to invest

Re: Vulnerability reports are not special anymore

#142

Post by inigyou »

ozim wrote: post #107
cleverfoo wrote: post #64

Earlier quoted context omitted.

It's not a horrible idea... the challenge there would be making that payment/refund flow totally transparent in order to build trust and be fair to the researchers.

Making, payment/refund setup is more complicated than „set and forget”. First question: Do you keep money for shit reports? Well no, you have to pay it back like credit card validation. There is no pain for posting shit report just inconvenience. There is no legal way where you can keep the money.

Why not?

Re: Vulnerability reports are not special anymore

#143

Post by _el1s7 »

> LLMs are as good as almost any security researcher, and anyone4 can run them.

What is this, rage bait? It's bullshit, and insulting to actual security researchers.

That might be true for low-effort vulnerabilities and fake security researchers, but the real security researchers are far from being replaced by LLMs.

Re: Vulnerability reports are not special anymore

#147

Post by jstimpfle »

cadamsdotcom wrote: post #4

Security through obscurity was never a great strategy.. and now it’s not a strategy at all.. Hopefully at the end of this decade, a ton of software practices have been overhauled to eliminate classes of problems. Memory-safe language use is a great start - but it’d be great to see innovation in checking for TOCTOU problems, improper/missing authn & authz, and many others. This is an engineering problem. It won’t be s…

It might just mean the opposite. Unergonomic and/or slow memory safe languages might not be needed anymore at some point, because LLM can check for reasonable programming patterns and can do some amount of vulnerability checking upfront. But seriously the first point -- as long as you keep to a known set of reasonable patterns (much larger set than the sets accepted by restrictive and cumbersome type systems), memory unsafe languages are actually pretty safe, and the code remains easy to understand, easy to maintain, and performant.

Already today, in my experience, widespread models like Claude 4.6 or 4.8 can quite reliably find some concurrency bugs that are easily missed by humans.

Re: Vulnerability reports are not special anymore

#148

Post by StrauXX »

jamesfinlayson wrote: post #62
cpuguy83 wrote: post #41

Earlier quoted context omitted.

In an ideal universe yes. But we live in a world where vulnerability scanners reign supreme.

Yep, I've updated dependencies with an RCE that can't be exploited in my codebase just to keep my security team happy. Not worth the multiple arguments about it not actually being an issue.

You can never guarantee that the codepath of a dependency that is vulnerable can not be reached or used as a gadget in an exploit chain. Patching dependencies, even when no direct vulnerability arises is an essential part of defense in depth and sevurity hygene.

Re: Vulnerability reports are not special anymore

#149

Post by ordu »

user3939382 wrote: post #10
cadamsdotcom wrote: post #4

Security through obscurity was never a great strategy.. and now it’s not a strategy at all.. Hopefully at the end of this decade, a ton of software practices have been overhauled to eliminate classes of problems. Memory-safe language use is a great start - but it’d be great to see innovation in checking for TOCTOU problems, improper/missing authn & authz, and many others. This is an engineering problem. It won’t be s…

Verifying correctness of an implementation is P NP, not serious CS research.

Rust's safety guarantees are also undecidable on an arbitrary program, but Rust still proves them for safe rust code. Why so? Because safe rust code is not an arbitrary program, it is specifically forbidden from doing some things, which becomes a pain in the ass sometimes for a coder, but the result is a subset of programs you can prove safety guarantees for.

The issue is how to design a language that restricts programmers in a way you can prove everything you care for on their programs, and still don't make life of programmers unbearably difficult, because the language allows them nothing.

Re: Vulnerability reports are not special anymore

#150

Post by throwaway2037 »

Quothling wrote: post #79
jamesfinlayson wrote: post #51

Earlier quoted context omitted.

I tried to raise that with my internal security team recently - don't clutter my vulnerability dashboard with issues in dev dependencies. They somewhat rightly pointed out that malware needs to be dealt even if it's a dev dependency. So my suggestion went nowhere because I guess we can't filter by type of vulnerability.

Working in the EU energy sector where we have to work with NIS2 compliance, I'd argue that your security team rightly pointed it out. I suspect that's what you mean though, and the rightly is just there because you agree with it but don't like it. We work with even more tight dependencies policies than just having alerts. We have a set of pre-approved and yearly vetted packages, like pandas or pyarrow for Python data…

First, this is great reply with lots of real world experience to share.

    > I do think external dependencies are among our biggest security threats though.
This sounds like a good business opportunity. I know that Sonatype has a business to vet Java dependencies. Does your company use it? I am guessing that Sonatype may be expanding into other open source ecosystems.
Post reply on HN