It's like supercharged fuzzing.
Hardening Firefox with Anthropic's Red Team
151–160 of 188 posts
Re: Hardening Firefox with Anthropic's Red Team
#152Earlier quoted context omitted.
To be clear, almost (all?) of mine do not either and it's partially due to the fact I have been really interested in formal methods thanks to Hillel Wayne, but I don't seem to have the math background for them. To the man who has seen a fancy new hammer but cannot afford it, every problem looks like a nail. The origin of it is a hypothesis I can get better quality code out of agents by making them do the things I don…
I guess my feeling is that formal verification _even in the LLM era_ still feels heavy-handed/too expensive for too little value for a lot of the problems I'm working on.
I guess I am trying to think laterally right now. There’s a lot of attention given to crafting the right prompt to get what you need, but I am a belt and suspenders kinda guy and my concern is even if we get it right the first time, what guarantee do I have I don’t ask for a change a year from now without thinking through the implications and it subtly breaks stuff. There’s basically zero cost to me currently to require formal verification, as long as we don’t count the oceans I am helping to boil.
Re: Hardening Firefox with Anthropic's Red Team
#153Re: Hardening Firefox with Anthropic's Red Team
#154Earlier quoted context omitted.
That sounds like the same problem (a deluge of slop) with a different interface (eating straight from the trough rather than waiting for someone to put a bow on it and stamp their name to it)?
I've found it's pretty good. It's really not that much of a burden to dig through 10 reports and find the 2 that are legitimate. It's different from Hacker One because those reports tend to come in with all sorts of flowery language added (or prompt-added) by people who don't know what they are doing. If you're running the prompts yourself against your own coding agents you gain much more control over the process. Yo…
You also probably have a much better idea of where the unsafe boundaries in your application are. Letting the models know this information up front has given me a dozen or so legitimate vulnerabilities in the application I work on. And the signal to noise ratio is generally pretty good. Certainly orders of magnitude better than the terrible dependabot alerts I have to dismiss every day
Re: Hardening Firefox with Anthropic's Red Team
#155I recommend that anyone who is responsible for maintaining the security of an open-source software project that they maintain ask Claude Code to do a security audit of it. I imagine that might not work that well for Firefox without a lot of care, because it's a huge project. But for most other projects, it probably only costs $3 worth of tokens. So you should assume the bad guys have already done it to your project l…
This is exactly how I would not recommend AI to be used. “do a thing that would take me a week” can not actually be done in seconds. It will provide results that resemble reality superficially. If you were to pass some module in and ask for finite checks on that, maybe. Despite the claims of agents… treat it more like an intern and you won’t be disappointed. Would you ask an intern to “do a security audit” of an enti…
It depends whether anyone was ever actually going to spend that week doing it the "hard" way. Having Claude do it in a few minutes beats doing nothing.
Put another way: I absolutely would have an intern work on a security audit. I would not have an intern replace a professional audit though.
It's otherwise a pretty low stakes use. I'd expect false positives to be pretty obvious to someone maintaining the code.
Re: Hardening Firefox with Anthropic's Red Team
#156It's cool that Mozilla updated https://www.mozilla.org/en-US/security/advisories/mfsa2026-1... because we were all wondering who had found 22 vulnerabilities in a single release (their findings were originally not attributed to anybody.)
Use After Free Use After Free Use After Free Use After Free Use After Free Use After Free Use After Free. I would be more satisfied if they gave a proper explanation of what these could have lead to rather than being "well maybe 0.001% chance to exploit this". They did vaguely go over how "two" exploits managed to drop a file, but how impactful is that? Dropping a file in abcd with custom contents in some folder rela…
You should generally assume that in a web browser any memory corruption bug can, when combined with enough other bugs and a lot of clever engineering, be turned into arbitrary code execution on your computer.
Re: Hardening Firefox with Anthropic's Red Team
#157Earlier quoted context omitted.
Use After Free Use After Free Use After Free Use After Free Use After Free Use After Free Use After Free. I would be more satisfied if they gave a proper explanation of what these could have lead to rather than being "well maybe 0.001% chance to exploit this". They did vaguely go over how "two" exploits managed to drop a file, but how impactful is that? Dropping a file in abcd with custom contents in some folder rela…
You should generally assume that in a web browser any memory corruption bug can, when combined with enough other bugs and a lot of clever engineering, be turned into arbitrary code execution on your computer.
The most important bit being the difficulty, AI finding 21 easily exploitable bugs is a lot more interesting than 21 that you need all the planets to align to work.
Re: Hardening Firefox with Anthropic's Red Team
#158Earlier quoted context omitted.
This is exactly how I would not recommend AI to be used. “do a thing that would take me a week” can not actually be done in seconds. It will provide results that resemble reality superficially. If you were to pass some module in and ask for finite checks on that, maybe. Despite the claims of agents… treat it more like an intern and you won’t be disappointed. Would you ask an intern to “do a security audit” of an enti…
It depends whether anyone was ever actually going to spend that week doing it the "hard" way. Having Claude do it in a few minutes beats doing nothing. Put another way: I absolutely would have an intern work on a security audit. I would not have an intern replace a professional audit though. It's otherwise a pretty low stakes use. I'd expect false positives to be pretty obvious to someone maintaining the code.
My point is that it’s one thing to say I want my intern to start doing a security audit.
It’s another thing to say hey intern security audit this entire code base.
LLM’s thrive on context. You need the right context at the right time, it doesn’t matter how good your model is if you don’t have that.
Re: Hardening Firefox with Anthropic's Red Team
#159I recommend that anyone who is responsible for maintaining the security of an open-source software project that they maintain ask Claude Code to do a security audit of it. I imagine that might not work that well for Firefox without a lot of care, because it's a huge project. But for most other projects, it probably only costs $3 worth of tokens. So you should assume the bad guys have already done it to your project l…
This is exactly how I would not recommend AI to be used. “do a thing that would take me a week” can not actually be done in seconds. It will provide results that resemble reality superficially. If you were to pass some module in and ask for finite checks on that, maybe. Despite the claims of agents… treat it more like an intern and you won’t be disappointed. Would you ask an intern to “do a security audit” of an enti…
> Would you ask an intern to “do a security audit” of an entire massive program?
Why not?
You can't relies solely on that, but having an extra pair of eye without prior assumption on the code always is good idea.
Re: Hardening Firefox with Anthropic's Red Team
#160Earlier quoted context omitted.
This is exactly how I would not recommend AI to be used. “do a thing that would take me a week” can not actually be done in seconds. It will provide results that resemble reality superficially. If you were to pass some module in and ask for finite checks on that, maybe. Despite the claims of agents… treat it more like an intern and you won’t be disappointed. Would you ask an intern to “do a security audit” of an enti…
IMO the key behavior is that LLMs are really good at fuzz testing, because they are probabilistic monkeys on typewriters that are much more code-aware than a conventional fuzz tester. They cannot produce a comprehensive security audit or fix security issues in a reliable way without human oversight, but they sure can come up with dumb inputs that break the code. The results of such AI fuzz testing should be treated a…
>the key behavior is that LLMs are really good at fuzz testing, because they are probabilistic monkeys on typewriters
That's exactly what they're not. Models post-trained with current methods/datasets have pretty poor diversity of outputs, and they're not that useful for fuzz testing unless you introduce input diversity (randomize the prompt), which is harder than it sounds because it has to be semantical. Pre-trained models have good output diversity, but they perform much worse. Poor diversity can be fixed in theory but I don't see any model devs caring much.