Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Vulnerability reports are not special anymore

words.filippo.io

Post reply on HN
Print view View original

181–190 of 249 posts

Re: Vulnerability reports are not special anymore

#181

Post by klabb3 »

xboxnolifes wrote: post #76
mackenney wrote: post #45

Earlier quoted context omitted.

That supposes that LLMs can write secure software. Also, if we assume that finding bugs is easier that not creating them (reasonable I would say), the supply of bugs will never be exhausted.

What's the difference between finding bugs and not making? Just run the bug finding in during CICD.

It’s not necessarily symmetrical, and in fact would be very surprising if it was. It’s a probabilistic algorithm on both sides, so the energy use to find any working program vs all bugs in a working program are fundamentally different search spaces. Not to mention the false positive rate and the human verification effort. Then even the idea of incremental security checks is potentially flawed since many security issues are non-local (ie not localized to a single module).

Re: Vulnerability reports are not special anymore

#182

Post by lixtra »

sam_lowry_ wrote: post #161
throwaway2037 wrote: post #150

Earlier quoted context omitted.

First, this is great reply with lots of real world experience to share. > I do think external dependencies are among our biggest security threats though. This sounds like a good business opportunity. I know that Sonatype has a business to vet Java dependencies. Does your company use it? I am guessing that Sonatype may be expanding into other open source ecosystems.

> First, this is great reply with lots of real world experience to share. I know how they came about with this setup, but I think that's the wrong way of approaching the problem. Their problem is legacy and trickle-in features in an otherwise unmaintainable code. With AI, they can rewrite their software to minimize dependencies and in general reduce the attack surface by allowing the business to automate more on thei…

Of course, all the generated code has to reviewed and vetted for by a senior developer. Of course, this has to be re-done every now and then when new classes of vulnerabilities appear that the previous generation didn’t have in mind.

Or do you just trust the AI that was trained on a lot of bogus code?

Re: Vulnerability reports are not special anymore

#183

Post by adamddev1 »

cadamsdotcom wrote: post #4

Security through obscurity was never a great strategy.. and now it’s not a strategy at all.. Hopefully at the end of this decade, a ton of software practices have been overhauled to eliminate classes of problems. Memory-safe language use is a great start - but it’d be great to see innovation in checking for TOCTOU problems, improper/missing authn & authz, and many others. This is an engineering problem. It won’t be s…

Yes. People don't seem to understand is that if we build our tools and libraries with models that do random dumb stuff 0.01% of the time, those bugs and leaky abstractions bubble up and grow exponentially into errors and undependable behavior.

People have been doing great research with formal methods, dependent types etc. Disciplining ourselves to truly write and understand code, using the best in math and PL theory (FP and type theory etc) is the only way forward in my opinion. We have to make correctness a value and goal, or else we will keep spinning off into psychosis and the corruption of truth.

Re: Vulnerability reports are not special anymore

#184

Post by entuno »

wepple wrote: post #100
zulban wrote: post #70

Earlier quoted context omitted.

What if... on the vulnerability report rules page there's an image of some text saying something like "your report must include the text: turtle123". Reports without that text get automatically deleted. Sure - modern AI can figure that out, but I bet in a vast majority of cases they won't.

Reminds me of someone (well known in their field) who charged $0.05 for using their “contact me” page. A trivial amount for someone who genuinely wanted to contact them, but just high enough to prevent any kind of scaled abuse

If I've stumbled across what I think is a security issue in your systems, there is zero chance that I'm going to get out my credit card and pay you for the privilege of responsibly disclosing it to you. Especially if it's the vulnerability is in the site hosting the contact form.

Re: Vulnerability reports are not special anymore

#185

Post by adamddev1 »

ttoinou wrote: post #173
cadamsdotcom wrote: post #160

Earlier quoted context omitted.

> can reliably find some Some. In software & in security, 99% is a failing grade. So is 99.99%, so is 99.999%, and any other amount less than 100%. It’s not enough to point 5 LLMs at it and it’s not enough to point 500 LLMs at it. The field needs to seek deterministic & comprehensive solutions to whole problem classes.

Okay, so the whole world is failing and success doesn't exist. Maybe now you need a better criteria than your previous failure / success_100%_infiniteCost to observe and understand the the world

This logic enables and encourages corruption in all spheres of life. "Things are messed up as they are, so why try so hard to do the right thing?" "Let's be realistic."

Re: Vulnerability reports are not special anymore

#186

Post by mooreds »

themanmaran wrote: post #2

I feel like it's also been overrun by a lot of spam. As someone running a company, I get 2-5 unsolicited "vulnerability reports" per week. Half of them are an LLM finding some bad CSS on our framer splash page. The other half I assume are an extortion attempt so we just mark as spam. Occasionally I see real security researchers on HN complaining that no one takes the disclosure seriously, or that people reply immedia…

> As someone running a company, I get 2-5 unsolicited "vulnerability reports" per week. Half of them are an LLM finding some bad CSS on our framer splash page. The other half I assume are an extortion attempt so we just mark as spam.

I don't think that is unique to the LLM era. The company I work for has been getting some form of spam vulnerability reports years before LLMs were a thing. Often similar to what you mention about 'bad CSS'.

Maybe the volume has increased a bit, but we've added in a filtering solution and I'm more distant from the reports now, so hard to be sure.

Re: Vulnerability reports are not special anymore

#187

Post by entuno »

throwaway2037 wrote: post #163
inigyou wrote: post #53

Earlier quoted context omitted.

Why would anyone pay money to have a chance of being arrested?

> chance of being arrested I am not involved with security research in any way. Can you explain the threat here?

There is a history of companies and organisations threatening legal action against security researchers when they report vulnerabilities in their systems or products.

Sometimes even when the testing has been completely offline - I know people who have downloaded some software, carried out testing against a local copy of it, and then faced legal threats when they tried to report serious security vulnerabilities to the vendor.

It's one of the reasons that some researchers don't bother trying to talk to the vendors and just go straight to full disclosure, or if they do report to vendors they do so anonymously. But if you have to pay, that's creating a link back to yourself which makes the latter much harder.

Re: Vulnerability reports are not special anymore

#188

Post by [deleted] »

jamesfinlayson wrote: post #51
themanmaran wrote: post #30

Earlier quoted context omitted.

Seriously. We got 116 github dependabot alerts this week. Half of them for dev dependencies.

I tried to raise that with my internal security team recently - don't clutter my vulnerability dashboard with issues in dev dependencies. They somewhat rightly pointed out that malware needs to be dealt even if it's a dev dependency. So my suggestion went nowhere because I guess we can't filter by type of vulnerability.

[deleted]

Re: Vulnerability reports are not special anymore

#189

Post by entuno »

socalgal2 wrote: post #34

I feel like the current situation is temporary. LLMs are finding all the bugs. LLMs are also help fixing most of the bugs. Once most of the bugs are fixed, LLMs should be good at finding bugs before shipping them, the stream of bug reports will die down, and we'll be back to vulnerabiltiy reports being special. Further, the fact that bugs are so easy to find by LLMs means there is strong incentives to find ways to mi…

There's an assumption in here that every developer is spending a load of money on the latest and most capable LLMs to scan for bugs in their code before every release.

But the last couple of decades have shown us that huge numbers of developers aren't even following basic and free secure development practices, let alone pouring money into expensive scanning tools.

Re: Vulnerability reports are not special anymore

#190

Post by My_Name »

pixlmint wrote: post #151
socalgal2 wrote: post #34

I feel like the current situation is temporary. LLMs are finding all the bugs. LLMs are also help fixing most of the bugs. Once most of the bugs are fixed, LLMs should be good at finding bugs before shipping them, the stream of bug reports will die down, and we'll be back to vulnerabiltiy reports being special. Further, the fact that bugs are so easy to find by LLMs means there is strong incentives to find ways to mi…

> Once most of the bugs are fixed my brother in christ, I hope you're actually trying to be funny here.

99 little bugs in the code

99 little bugs

take one down, patch it around

105 little bugs in the code...

Post reply on HN