Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardening Firefox with Anthropic's Red Team

anthropic.com

Post reply on HN
Print view View original

51–60 of 188 posts

Re: Hardening Firefox with Anthropic's Red Team

#51

Post by est31 »

I suppose eventually we'll see something like Google's OSS-Fuzz for core open source projects, maybe replacing bug bounty programs a bit. Anthropic already hands out Claude access for free to OSS maintainers.

LLMs made it harder to run bug bounty programs where anyone can submit stuff, and where a lot of people flooded them with seemingly well-written but ultimately wrong reports.

On the other hand, the newest generation of these LLMs (in their top configuration) finally understands the problem domain well enough to identify legitimate issues.

I think a lot of judging of LLMs happens on the free and cheaper tiers, and quality on those tiers is indeed bad. If you set up a bug bounty program, you'll necessarily get bad quality reports (as cost of submission is 0 usually).

On the other hand, if instead of a bug bounty program you have an "top tier LLM bug searching program", then then the quality bar can be ensured, and maintainers will be getting high quality reports.

Maybe one can save bug bounty programs by requiring a fee to be paid, idk, or by using LLM there, too.

Re: Hardening Firefox with Anthropic's Red Team

#52

Post by suddenlybananas »

est31 wrote: post #51

I suppose eventually we'll see something like Google's OSS-Fuzz for core open source projects, maybe replacing bug bounty programs a bit. Anthropic already hands out Claude access for free to OSS maintainers. LLMs made it harder to run bug bounty programs where anyone can submit stuff, and where a lot of people flooded them with seemingly well-written but ultimately wrong reports. On the other hand, the newest genera…

> Anthropic already hands out Claude access for free to OSS maintainers.

Free for 6 months after which it auto-renews if I recall correctly.

Re: Hardening Firefox with Anthropic's Red Team

#53

Post by amelius »

mozdeco wrote: post #40
amelius wrote: post #38

Perhaps I missed it but I don't see any false positives mentioned.

[working for Mozilla] That's because there were none. All bugs came with verifiable testcases (crash tests) that crashed the browser or the JS shell. For the JS shell, similar to fuzzing, a small fraction of these bugs were bugs in the shell itself (i.e. testing only) - but according to our fuzzing guidelines, these are not false positives and they will also be fixed.

Sounds good.

Did you also test on old source code, to see if it could find the vulnerabilities that were already discovered by humans?

Re: Hardening Firefox with Anthropic's Red Team

#54

Post by sigmar »

est31 wrote: post #51

I suppose eventually we'll see something like Google's OSS-Fuzz for core open source projects, maybe replacing bug bounty programs a bit. Anthropic already hands out Claude access for free to OSS maintainers. LLMs made it harder to run bug bounty programs where anyone can submit stuff, and where a lot of people flooded them with seemingly well-written but ultimately wrong reports. On the other hand, the newest genera…

>where a lot of people flooded them with seemingly well-written but ultimately wrong reports.

are there any projects to auto-verify submitted bug reports? perhaps by spinning up a VM and then having an agent attempt to reproduce the bug report? that would be neat.

Re: Hardening Firefox with Anthropic's Red Team

#55

Post by tptacek »

pjmlp wrote: post #5
fcpk wrote: post #2

The fact there is no mention of what were the bugs is a little odd. It'd really be nice to see if this is a "weird never happening edge case" or actual issues. LLMs have uncanny abilities to identify failure patterns that it has seen before, but they are not necessarily meaningful.

Indeed, without it looks like a fluffy marketing piece.

And now that you know that it isn't, do you feel differently about the logic you used to write this comment?

Re: Hardening Firefox with Anthropic's Red Team

#56

Post by Quarrel »

amelius wrote: post #53
mozdeco wrote: post #40

Earlier quoted context omitted.

[working for Mozilla] That's because there were none. All bugs came with verifiable testcases (crash tests) that crashed the browser or the JS shell. For the JS shell, similar to fuzzing, a small fraction of these bugs were bugs in the shell itself (i.e. testing only) - but according to our fuzzing guidelines, these are not false positives and they will also be fixed.

Sounds good. Did you also test on old source code, to see if it could find the vulnerabilities that were already discovered by humans?

I really like this as a suggestion, but getting opensource code that isn't in the LLMs training data is a challenge.

Then, with each model having a different training epoch, you end up with no useful comparison, to decide if new models are improving the situation. I don't doubt they are, just not sure this is a way to show it.

Re: Hardening Firefox with Anthropic's Red Team

#57

Post by hu3 »

rs_rs_rs_rs_rs wrote: post #43
vorticalbox wrote: post #31

Earlier quoted context omitted.

its just a different attack surface for safari they would need to blackbox attack the browser which is much harder than what they did her

What? The js engine in Safari is open source, they can put Claude to work on it any time they want.

There's much more to a browser than JS engine.

They picked to most open-source one.

Re: Hardening Firefox with Anthropic's Red Team

#58

Post by utopiah »

simonw wrote: post #44
lloydatkinson wrote: post #15

Anthropic feels like they are flailing around constantly trying to find something to do. A C compiler that didn't work, a browser that didn't work, and now solving bugs in Firefox.

What was Anthropic's "browser that didn't work"?

I think they meant Cursor, cf https://news.ycombinator.com/item?id=46646777

Re: Hardening Firefox with Anthropic's Red Team

#59

Post by ferguess_k »

lloydatkinson wrote: post #15

Anthropic feels like they are flailing around constantly trying to find something to do. A C compiler that didn't work, a browser that didn't work, and now solving bugs in Firefox.

However, the shape is there. And no one knows how good the thing is going to be after X months. We are measuring months here, not even years.

I believe there is a theoretical cap about the capability of LLM. I'm wondering what does it look like.

Re: Hardening Firefox with Anthropic's Red Team

#60

Post by utopiah »

nz wrote: post #26
mentalgear wrote: post #3

That's one good use of LLMs: fuzzy testing / attack.

Not contradicting this (I am sure it's true), but why is using an LLM for this qualitatively better than using an actual fuzzer?

I didn't even read the piece but my bet is that fuzzers are typically limited to inputs whereas relying on LLMs is also about find text patterns, and a bit more loosely than before while still being statistically relevant, in the code base.
Post reply on HN