Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability reports are not special anymore

words.filippo.io

Post reply on HN
Print view View original

211–220 of 249 posts

Re: Vulnerability reports are not special anymore

#211

Post by bluGill »

wongarsu wrote: post #141
socalgal2 wrote: post #92

Earlier quoted context omitted.

If LLMs can trivially find bugs, then they can trivially find bugs. If they can't find any bugs that doesn't mean there are no bugs but it suggests that others can't easily find them either. So the LLMs find all the bugs problem is fixed by asking the LLMs to find them before you ship. Read what wrote, I didn't say your program will be bug free. I said, if the LLM can trivally find the bug it will. If it can't then w…

The bugs I can trivially find, the bugs Claude can trivially find and the bugs Codex can trivially find are not necessarily the same. The most obvious bugs would be obvious to any of us three, but beyond that we would't agree on a definition of 'trivial bug'. This gets worse if you factor in different harnesses built for the task, and future model updates Sure, things will get calmer than they are now, but shipping w…

But by using a LLM everyone is automatically devoting that much effort, and so while we don't fix all bugs we fix a lot that wouldn't have been.

Re: Vulnerability reports are not special anymore

#212

Post by parasense »

Can anybody say what is going to happen? It's not a rhetorical question, and the implied or entailed context might involve a Nash equilibrium of some sort.

Right now the rate of signal is high, and the ratio of noise is proportionally high. But it seems like everyone expects the signal to eventually plateau or sharply decline. Almost as if there is a finite supply of "low hanging fruit" for shallow scanning machines to easily discover, and then there will be some kind of new world that follows where only truly difficult problems emerge.

But eventually the question then becomes why even bother with Rust or any other silly borrow checking ideas if we can use more enjoyable programming languages with LLM side-kicks to catch security vulnerabilities on the front side of the development workflow?

IT seems to me if we exhaust all the extant security vulnerabilities to a calculus that asymptotically goes to infinitesimal zeroness, then... the only trick remaining is to scan code before it becomes vulnerable.

Re: Vulnerability reports are not special anymore

#213

Post by bluGill »

aetherspawn wrote: post #117

LLMs find more vulnerabilities than people because people time is heaps more expensive than LLM time, that’s it. We’ve always been able to find heaps, we’ve just never had the right structures to put in the effort and renumerate people for looking (even if they don’t find anything).

Looking for bugs is boring. LLMs don't get bored the same way humans do. Thus even if you had infinite budget you should expect a LLM to be better.

Re: Vulnerability reports are not special anymore

#214

Post by notpushkin »

ozim wrote: post #175
inigyou wrote: post #142

Earlier quoted context omitted.

Why not?

Because you are not providing any service not selling anything. There is no real way as a company to withhold someone’s money and that it goes through accounting. I am not an accountant so ask some accountants why not.

To participate in the bug bounty program, you must pay ACME Inc. $1 (one U.S. dollar) per submission. This payment is non-refundable as it covers our triage costs and bounty payment processing fees. You may submit a vulnerability without paying, but you will not be eligible for receiving any bounty payments under this program.

If your disclosure otherwise meets all of the guidelines of the program, but is not eligible for a bounty, we may, in our sole discretion, award you a bounty of $1.

Re: Vulnerability reports are not special anymore

#215

Post by Goofy_Coyote »

Security Eng here. The whole thing is an absolute mess. I’ve been (and still am) on both sides of the fence.

I currently have two reports (one RCE on a famous OSS ML platform, one cluster take over on a k8s related projects), both are more than 2 months old without as much as an “F you, get lost”. Just got ignored and ghosted, which hurts a lot, because I spent a lot of time finding, and verifying these (all reports with poc and patch). BUT I understand why it’s happening, because I’m also on the receiving end.

security@ and VDPs have always received BS reports and beg-bounties, but boy oh boy, these days we have two people spending 3-4 days a week sifting through this constant flood of garbage compared to 2-3 tears ago where 1 person could triage the inbox and VDP in a day’s work max, which would’ve been considered very busy. Unfortunately we can’t just shutdown the programs or the mailbox because 1. We do occasionally get important and great stuff that actually matters, and 2. We’re a critical infra company and can’t ignore anything really.

The signal to noise ratio is almost zero, but the “what if” is keeping us swimming through this unending river of garbage and burning us out.

Overall, chaotic mess on both sides.

Ending on a doom-and-gloom note: there will be a reckoning.

(Don’t take the note too seriously though, I’m a SecEng, so I have a built-in doom multiplier lol)

Re: Vulnerability reports are not special anymore

#216

Post by AlotOfReading »

Maxion wrote: post #109
Gigachad wrote: post #90

Earlier quoted context omitted.

It's easier to keep stuff up to date these days. If you have a project with typescript, unit tests, and end to end tests like cypress you can just have dependabot create the PRs to update packages. If everything passes you just have to hit the merge button. Just updating everything is probably easier than assessing if it's possible to trigger an exploit with the way you use the package.

Yep this is what has happened to small teams. You really only have time to approve the dependabot changes and go go go. Otherwise you'll never get anything productive done. The other option is to simply ignore updates and do them on a schedule, e.g. once every 1-2 months.

Or you take the alternative approach of flattening and minimizing your dependency graph. Having so many dependencies you can't reasonably field bug reports in them is a chosen tradeoff, even if it doesn't feel that way.

Re: Vulnerability reports are not special anymore

#217

Post by ezekg »

themanmaran wrote: post #30
Gigachad wrote: post #6

Earlier quoted context omitted.

I'm getting CVE fatigue with all of these super ultra critical 10/10 vulnerabilities that are some node package that compiles my frontend can get stuck if I give it a malicious regex. It's hard to spot the stuff that actually matters.

Seriously. We got 116 github dependabot alerts this week. Half of them for dev dependencies.

It's even worse if you're SOC2 because then you actually have to go through and mark them as "not exploitable." The noise is insane right now.

Re: Vulnerability reports are not special anymore

#218

Post by Quothling »

throwaway2037 wrote: post #150
Quothling wrote: post #79

Earlier quoted context omitted.

Working in the EU energy sector where we have to work with NIS2 compliance, I'd argue that your security team rightly pointed it out. I suspect that's what you mean though, and the rightly is just there because you agree with it but don't like it. We work with even more tight dependencies policies than just having alerts. We have a set of pre-approved and yearly vetted packages, like pandas or pyarrow for Python data…

First, this is great reply with lots of real world experience to share. > I do think external dependencies are among our biggest security threats though. This sounds like a good business opportunity. I know that Sonatype has a business to vet Java dependencies. Does your company use it? I am guessing that Sonatype may be expanding into other open source ecosystems.

Companies like Sonatype would be an issue since they are owned by USA private equity. We would not give "Vista Equity" access to anything with the current EU US relationship. It's bad enough that we're so tied into Microsoft, which the EU might task us with leaving if they deem it critical enough for the security of the European energy sector. That's a risk we live with though, there isn't a realistic alternative.

That being said, our current strategy is more along the lines of building thind within standard libraries. We really wanted to adopt Go company wide, but it's proven impossible for non-SWE staff to use AI to create their projects in anything but Python. So instead we've created AI configurations that know our security policies, the tools we want them to use and we've setup security policies which won't even allow you to run a Python executionable inside a virtual environment unless your devices is sepcifically allowed to do so in that specific folder. Similarily we've completely limited what VSCode extensions they can use down to the named folder version. Which sort of sucks, and I doubt a lot of it would be possible if it wasn't because the c-levels are personally liable for security under EU law.

We'll see what happens after september when the summer holidays are over and the real token cost of AI will kick in.

Re: Vulnerability reports are not special anymore

#219

Post by PaulStatezny »

Goofy_Coyote wrote: post #215

Security Eng here. The whole thing is an absolute mess. I’ve been (and still am) on both sides of the fence. I currently have two reports (one RCE on a famous OSS ML platform, one cluster take over on a k8s related projects), both are more than 2 months old without as much as an “F you, get lost”. Just got ignored and ghosted, which hurts a lot, because I spent a lot of time finding, and verifying these (all reports…

> Ending on a doom-and-gloom note: there will be a reckoning.

Can you elaborate on what you mean by this?

Post reply on HN