[dead]
Hardening Firefox with Anthropic's Red Team
161–170 of 188 posts
Re: Hardening Firefox with Anthropic's Red Team
#162Earlier quoted context omitted.
Seems like a poor advertisement for their product if their shining example of utility is a broken compiler that doesn't function as the README indicates.
Impressive that it made a c compiler though? Or do we judge all programmers by their documentation now?
All it took was all the C compilers they could scrape into their training set.
It’s not impressive in the sense that it’s doing what it was designed to.
It just happens that it generated a C compiler that kind of worked.
Someone came by later and used more AI on it to make it closer to a production grade C compiler like gcc/clang.
Saying, “it made a C compiler,” is not specific enough.
Re: Hardening Firefox with Anthropic's Red Team
#163I recommend that anyone who is responsible for maintaining the security of an open-source software project that they maintain ask Claude Code to do a security audit of it. I imagine that might not work that well for Firefox without a lot of care, because it's a huge project. But for most other projects, it probably only costs $3 worth of tokens. So you should assume the bad guys have already done it to your project l…
This is exactly how I would not recommend AI to be used. “do a thing that would take me a week” can not actually be done in seconds. It will provide results that resemble reality superficially. If you were to pass some module in and ask for finite checks on that, maybe. Despite the claims of agents… treat it more like an intern and you won’t be disappointed. Would you ask an intern to “do a security audit” of an enti…
What is there to loose in trying?
Basically, don't trust AI if it says "you program is secure", but if it returns results how you could break it, why not take a look?
This is the way I would encourage AI to be used, I prefer such approaches (e.g. general code reviews) than writing software by it.
Re: Hardening Firefox with Anthropic's Red Team
#164Earlier quoted context omitted.
The HackerOne slop is because there's a financial incentive (bug bounties) involved, which means people who don't know what they are doing blindly submit anything that an LLM spots for them. If you're running the security audit yourself you should be in a better position to understand and then confirm the issues that the coding agents highlight. Don't treat something as a security issue until you can confirm that it…
That sounds like the same problem (a deluge of slop) with a different interface (eating straight from the trough rather than waiting for someone to put a bow on it and stamp their name to it)?
Seems very similar to turning on compiler warnings. A load of scary nothings, and a few bugs. But you fix the bugs and clarify the false positives, and end up with more robust and maintainable code.
Re: Hardening Firefox with Anthropic's Red Team
#165The fact there is no mention of what were the bugs is a little odd. It'd really be nice to see if this is a "weird never happening edge case" or actual issues. LLMs have uncanny abilities to identify failure patterns that it has seen before, but they are not necessarily meaningful.
[deleted]
Re: Hardening Firefox with Anthropic's Red Team
#166"But it was still unclear how much we should trust this result because it was possible that at least some of those historical CVEs were already in Claude’s training data." I feel like they could know this if they truly wanted to. It's honestly unnerving that an AI company cant say for certain if their models were trained on something.
Re: Hardening Firefox with Anthropic's Red Team
#167Earlier quoted context omitted.
[work at Mozilla] I agree that LLMs are sometimes wrong, which is why this new method here is so valuable - it provides us with easily verifiable testcases rather than just some kind of analysis that could be right or wrong. Purely triaging through vulnerability reports that are static (i.e. no actual PoC) is very time consuming and false-positive prone (same issue with pure static analysis). I can't really confirm t…
Please, implement "name window" natively in Firefox. I have to use chrome because the lack of it.
Re: Hardening Firefox with Anthropic's Red Team
#168This seems like a win for open source maintainers pressed on time and resources. Whether or not LLMs find novel security risks or just pattern-match known issues, many vulnerabilities are discovered late (or never) simply because nobody has the bandwidth to audit every file.
Re: Hardening Firefox with Anthropic's Red Team
#169I've had mixed results. I find that agents can be great for: 1. Producing new tests to increase coverage. Migrating you to property testing. Setting up fuzzing. Setting up more static analysis tooling. All of that would normally take "time" but now it's a background task. 2. They can find some vulnerabilities. They are "okay" at this, but if you are willing to burn tokens then it's fine. 3. They are absolutely wrong…
[work at Mozilla] I agree that LLMs are sometimes wrong, which is why this new method here is so valuable - it provides us with easily verifiable testcases rather than just some kind of analysis that could be right or wrong. Purely triaging through vulnerability reports that are static (i.e. no actual PoC) is very time consuming and false-positive prone (same issue with pure static analysis). I can't really confirm t…
My point is that "verifiable testcases" works great for proving "this is vulnerable" but LLMs are still risky if you believe "this is safe", which you can't easily prove. My point is that you need to be very skeptical of when they decide that something isn't vulnerable.
I completely agree that LLMs are great when instructed to provide provable, repeatable exploits. I have done this multiple times and uncovered some neat bugs.
> I can't really confirm the part about "local" bugs anymore though, but that might also be a model thing.
I don't think it's a model thing, it's just a sort of basic limitation of the technology. We shouldn't expect LLMs to perform novel tasks so we shouldn't expect LLMs to find novel vulnerabilities.
Agents help, human in the loop is critical for "injecting novelty" as I put it. The LLM becomes great at producing POCs to test out.
Re: Hardening Firefox with Anthropic's Red Team
#170I've had mixed results. I find that agents can be great for: 1. Producing new tests to increase coverage. Migrating you to property testing. Setting up fuzzing. Setting up more static analysis tooling. All of that would normally take "time" but now it's a background task. 2. They can find some vulnerabilities. They are "okay" at this, but if you are willing to burn tokens then it's fine. 3. They are absolutely wrong…
I've seen fairly poor results from people asking AI agents to fill in coverage holes. Too many tests that either don't make sense, or add coverage without meaningfully testing anything. If you're already at a very high coverage, the remaining bits are presumably just inherently difficult.
I suppose it's mixed results but a coverage report should give you "these exact lines are uncovered" and it becomes pretty straightforward to see "ah yeah that error condition isn't tracked, the behavior should be X, go write that test".