Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardening Firefox with Claude Mythos Preview

hacks.mozilla.org

Post reply on HN
Print view View original

31–40 of 176 posts

Re: Hardening Firefox with Claude Mythos Preview

#31

Post by IainIreland »

paulvnickerson wrote: post #23
IainIreland wrote: post #21

Earlier quoted context omitted.

I work at Mozilla; I fixed a bunch of these bugs. In general, I would say that our use of "vulnerability" lines up with what jerrythegerbil calls "potential vulnerability". (In cases with a POC, we would likely use the word "exploit".) Our goal is to keep Firefox secure. Once it's clear that a particular bug might be exploitable, it's usually not worth a lot of engineering effort to investigate further; we just fix i…

What types of vulnerabilities was it finding? Cross site scripting, privilege escalation, etc? Mostly memory corruption or any Javascript logic bugs?

I work on SpiderMonkey, so I mostly looked at the JS bugs. It was a smorgasbord of various things. Broadly speaking I'd say the most impressive bugs were TOCTOU issues, where we checked something and later acted on it, and the testcase found a clever way to invalidate the result of the check in between.

If you look closely at, say, this patch, you might get a sense of what I mean (although the real cleverness is in the testcase, which we have not made public): https://hg-edge.mozilla.org/integration/autoland/rev/c29515d...

Re: Hardening Firefox with Claude Mythos Preview

#32

Post by reisse »

IainIreland wrote: post #31
paulvnickerson wrote: post #23

Earlier quoted context omitted.

What types of vulnerabilities was it finding? Cross site scripting, privilege escalation, etc? Mostly memory corruption or any Javascript logic bugs?

I work on SpiderMonkey, so I mostly looked at the JS bugs. It was a smorgasbord of various things. Broadly speaking I'd say the most impressive bugs were TOCTOU issues, where we checked something and later acted on it, and the testcase found a clever way to invalidate the result of the check in between. If you look closely at, say, this patch, you might get a sense of what I mean (although the real cleverness is in t…

> although the real cleverness is in the testcase, which we have not made public

What is the point of keeping it private? I'd bet feeding this patch to Opus and asking to look for specific TOCTOU issue fixed by the patch will make it come up with a testcase sooner or later.

Re: Hardening Firefox with Claude Mythos Preview

#33

Post by paulvnickerson »

IainIreland wrote: post #31
paulvnickerson wrote: post #23

Earlier quoted context omitted.

What types of vulnerabilities was it finding? Cross site scripting, privilege escalation, etc? Mostly memory corruption or any Javascript logic bugs?

I work on SpiderMonkey, so I mostly looked at the JS bugs. It was a smorgasbord of various things. Broadly speaking I'd say the most impressive bugs were TOCTOU issues, where we checked something and later acted on it, and the testcase found a clever way to invalidate the result of the check in between. If you look closely at, say, this patch, you might get a sense of what I mean (although the real cleverness is in t…

Very cool, thank you.

Re: Hardening Firefox with Claude Mythos Preview

#34

Post by mozdeco »

jerrythegerbil wrote: post #12

Again, and this is important: A bug is a bug. A “potential vulnerability” is a bug. A vulnerability is verifiable as having security implications with a proof of concept or other substantial evidence. Words matter. Bugs matter. It’s important to fix large amounts of bugs, just as it always has been, and has been done. Let that be impressive on its own, because it IS impressive. Mythos didn’t write 271 PoC for vulnera…

Mythos did in fact write PoCs for all bugs that crash with demonstration of memory-unsafe behavior (e.g. use-after-free, out-of-bounds reads/writes, etc).

For us this is substantial enough evidence to consider it a security vulnerability at that point, unless shown otherwise and it has always been this way (also for fuzzing bugs).

Re: Hardening Firefox with Claude Mythos Preview

#35

Post by MetaverseClub »

rockdoe wrote: post #17
canucker2016 wrote: post #15

Earlier quoted context omitted.

Coverity (similar to lint) scans various open source software products for vulnerabilities. see https://www.blackduck.com/static-analysis-tools-sast/coverit... and for Firefox-related alleged defects, see https://scan.coverity.com/projects/firefox You have to create an account to view the actual reported defects. There are just over 5000 reported defects still outstanding. I don't know how many overlap with the repor…

How many of those are false positives though? Probably just over 5000? You get bug bounties if you report the kind of bugs Mythos identified. There's a reason no-one collected bounties from the "5000 defects" Coverity identified. The Mythos reports have several examples of chaining a whole bunch of logic in different parts of the program together to exploit something very subtle. The Coverity reports aren't anything…

it's just sad that Coverity represents the best working C++ static analysis tool.

Re: Hardening Firefox with Claude Mythos Preview

#36

Post by mmooss »

> “That’s the key thing that has unlocked our ability to operate at the scale we’ve been operating at now,” he said. “It gives the engineer a crank they can pull that says: ‘Yep, this has the problem,’ and then you can iterate on the code and know clearly when you’ve fixed it and eventually land the test case in the tree such that you don’t regress it.”

I don't understand much of this paragraph:

* "a crank they can pull that says: ‘Yep, this has the problem,’": as in, ring an alarm? Does the LLM ring th alarm?

* "you can iterate on the code and know clearly when you’ve fixed it": Isn't that true of most bugs, assuming you do the normal thing and generate a test case? And I thought the LLM output test cases itself: "It will craft test cases. We have our existing fuzzing systems and tools to be able to run those tests" And are they claiming the LLM facilitates iterating?

* "and eventually land the test case in the tree": Don't you create the test case before the fix? And just a few words earlier they seemed to be working on the fix, not the test case. And see the prior point about test cases.

* "such that you don’t regress it.”: How is the LLM helping here?

Maybe I'm missing some fundamental unwritten assumption?

Re: Hardening Firefox with Claude Mythos Preview

#37

Post by rem1099 »

I don't find that number very high. In a project of the size of Firefox, a new version of a compiler with stricter warnings or a draconian interpretation of the C standard can easily find 200 new bugs.

New tools find new bugs, but the oligarchy newspapers report on Mythos and not on clang-22.0.

Re: Hardening Firefox with Claude Mythos Preview

#38

Post by mccr8 »

paulvnickerson wrote: post #23
IainIreland wrote: post #21

Earlier quoted context omitted.

I work at Mozilla; I fixed a bunch of these bugs. In general, I would say that our use of "vulnerability" lines up with what jerrythegerbil calls "potential vulnerability". (In cases with a POC, we would likely use the word "exploit".) Our goal is to keep Firefox secure. Once it's clear that a particular bug might be exploitable, it's usually not worth a lot of engineering effort to investigate further; we just fix i…

What types of vulnerabilities was it finding? Cross site scripting, privilege escalation, etc? Mostly memory corruption or any Javascript logic bugs?

I'd say it leans towards memory corruption kinds of issues, as those are easiest to pass the validator, thanks to AddressSanitizer. I think there's a lot of potential for making the validator more sophisticated. Like maybe you add a JS function that will only crash when run in the parent process and have a validator that checks for that specific crash, as a way for the LLM to "prove" that it managed to run arbitrary JS in the parent. Would that turn up subtler issues? Maybe.

Re: Hardening Firefox with Claude Mythos Preview

#39

Post by mccr8 »

reisse wrote: post #32
IainIreland wrote: post #31

Earlier quoted context omitted.

I work on SpiderMonkey, so I mostly looked at the JS bugs. It was a smorgasbord of various things. Broadly speaking I'd say the most impressive bugs were TOCTOU issues, where we checked something and later acted on it, and the testcase found a clever way to invalidate the result of the check in between. If you look closely at, say, this patch, you might get a sense of what I mean (although the real cleverness is in t…

> although the real cleverness is in the testcase, which we have not made public What is the point of keeping it private? I'd bet feeding this patch to Opus and asking to look for specific TOCTOU issue fixed by the patch will make it come up with a testcase sooner or later.

Possibly! One of the many areas that might need rethinking in the age of AI (that started in February of this year) is how long security bugs should be hidden. We live in interesting times.

Re: Hardening Firefox with Claude Mythos Preview

#40

Post by tialaramex »

They've only linked a few tickets, so of course maybe when we see all 271 actual distinct things the insight won't apply but all those I examined ended up as some C++ code with a nasty bug in it.

Firefox is written in several languages, only about 25% of it is in C++ but every single one of these issues seems to touch the C++.

Post reply on HN