Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on the Recent Firefox Add-On Outage

hacks.mozilla.org

Post reply on HN
Print view View original

141–150 of 279 posts

Re: Technical Details on the Recent Firefox Add-On Outage

#141

Post by brokenmachine »

DangerousPie wrote: post #51

My point of view as a long-time Firefox user that cares about privacy but also knows we live in an imperfect world: It obviously sucks that this happened but I think they handled it very well. The bug was fixed so quickly that I wouldn't even have realized it had happened if it hadn't been for the thread here on HN. My extensions hadn't even been disabled yet by the time the patch came out. And pushing out the hotfix…

I had a totally different experience.

I'm still running FF 56.0.2 because I can't live without Tab Mix Plus.

The official line was to wait for the update but about:studies never came up with anything even after 24 hours of waiting and everyone was saying it was fixed but it wasn't for me, I presume because nobody cares about the refugees stuck on pre-add-on breaking versions.

So it was completely broken until I finally found a reddit thread that described how to use the developer console to manually import the certificate extracted from the fix.

That worked, but it enabled all my add-ons, even ones that were previously disabled.

Bloody irritating. I didn't even know that it was possible to break things remotely like this.

Re: Technical Details on the Recent Firefox Add-On Outage

#142

Post by y-satellite »

brokenmachine wrote: post #141
DangerousPie wrote: post #51

My point of view as a long-time Firefox user that cares about privacy but also knows we live in an imperfect world: It obviously sucks that this happened but I think they handled it very well. The bug was fixed so quickly that I wouldn't even have realized it had happened if it hadn't been for the thread here on HN. My extensions hadn't even been disabled yet by the time the patch came out. And pushing out the hotfix…

I had a totally different experience. I'm still running FF 56.0.2 because I can't live without Tab Mix Plus. The official line was to wait for the update but about:studies never came up with anything even after 24 hours of waiting and everyone was saying it was fixed but it wasn't for me, I presume because nobody cares about the refugees stuck on pre-add-on breaking versions. So it was completely broken until I final…

You realize, I hope, that 56.0.2 is riddled with security holes at this point? I get the attachment to old addons, but 129 CVEs (including multiple severe memory corruption bugs) affect that version now. It's not really reasonable to expect Mozilla to keep maintaining it.

Re: Technical Details on the Recent Firefox Add-On Outage

#144

Post by spectramax »

sohkamyung wrote: post #137

Related to this: Mozilla has deleted Telemetry data for those users who enabled Telemetry to get the hot-fix [1] [1] https://twitter.com/firefox/status/1126593558490693632

I know Mozilla is not your typical for-profit organization, but it is so nice to see "We failed and we are sorry." It felt good. This post-mortem read like a genuine story of what happened and how they solved the problem. No lawyer language. No bullshit.

I love Mozilla.

Re: Technical Details on the Recent Firefox Add-On Outage

#145

Post by yellowapple »

callahad wrote: post #130
yellowapple wrote: post #117

Earlier quoted context omitted.

I know full well it's not actually the case, but that doesn't make it not feel like it could be the case. It feels scummy, and I'd expect Mozilla to be above that scumminess. Like, just link to the XPI. Not that hard. The unexplained reluctance to do so is suspicious.

Gotcha. Manually installing the hotfix XPI makes cleanup a bit harder now that we have a proper fix. E.g., without coming from Studies, there's no study to ever end . Direct installation also makes it harder to quickly respond to any bugs we might discover in the initial revision of the hotfix. Now that we have a stable fix, we will publish an XPI with the option of direct installation for users of older, unsupported…

I see. Some follow up questions:

> Manually installing the hotfix XPI makes cleanup a bit harder now that we have a proper fix. E.g., without coming from Studies, there's no study to ever end.

The language around enabling Studies for the hotfix also claimed that once the hotfix installed, one can feel free to turn off Studies. Could similar language not have been included for the XPI approach (e.g. "once the fix is applied, you can uninstall this add-on")? Or is this a case where the extension does have to be installed (at least until the user upgrades to a point release with a fixed certificate)?

Alternately, do extensions have the ability to uninstall themselves? If so, then perhaps the extension could install the new certificate and immediately uninstall itself (or, in the "extension has to be installed for the fix to exist" scenario above, uninstall itself if it detects itself running on an updated Firefox and/or flag itself as incompatible with Firefoxen newer than the latest affected version)?

Alternately, is there no way for Firefox itself (e.g. in a point release) to explicitly blacklist an extension?

Alternately, is it possible to revoke the certificate/signature for that extension such that Firefox deems it invalid and disables it (using, presumably, the same mechanism and rationale as what caused this particular bug)?

Seems like this is a problem with multiple potential solutions besides "just do it as a Study". Even if it really is/was unsolvable, I feel like power users would be perfectly happy with getting the quick fix in exchange for subsequent cleanup being on them; ain't ideal, but it's better than waiting for multiple hours for Studies to work its magic.

> Direct installation also makes it harder to quickly respond to any bugs we might discover in the initial revision of the hotfix.

I'm sure there are some people out there who would be happy to test the XPI while having Telemetry enabled so y'all can get all that juicy fresh debugging data :)

Re: Technical Details on the Recent Firefox Add-On Outage

#146

Post by lifthrasiir »

ekianjo wrote: post #138

I love how they paint the picture that certificates "unfortunately expired" as if it were an act of god or something. Surely one cannot see it coming! No mention at all why nobody was checking certificates expiry.

The post seems to imply that it was a simple overlook (which is frequent when every such thing is not formally tracked). I agree that it is hard.

> We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make, but in the meantime here are my initial thoughts about what we need to do. First, we should have a much better way of tracking the status of everything in Firefox that is a potential time bomb and making sure that we don’t find ourselves in a situation where one goes off unexpectedly. We’re still working out the details here, but at minimum we need to inventory everything of this nature.

Re: Technical Details on the Recent Firefox Add-On Outage

#147

Post by brokenmachine »

y-satellite wrote: post #142
brokenmachine wrote: post #141

Earlier quoted context omitted.

I had a totally different experience. I'm still running FF 56.0.2 because I can't live without Tab Mix Plus. The official line was to wait for the update but about:studies never came up with anything even after 24 hours of waiting and everyone was saying it was fixed but it wasn't for me, I presume because nobody cares about the refugees stuck on pre-add-on breaking versions. So it was completely broken until I final…

You realize, I hope, that 56.0.2 is riddled with security holes at this point? I get the attachment to old addons, but 129 CVEs (including multiple severe memory corruption bugs) affect that version now. It's not really reasonable to expect Mozilla to keep maintaining it.

I don't expect them to maintain it, but I do expect them not to break it for no reason.

Also they could have mentioned in their post that their fix did not do anything for older versions, instead of specifically telling everyone to just keep waiting.

xpinstall.signatures.required = false didn't fix it.

I'll be very happy to update when there's a version that has a good tab manager. I'm on the latest version at home and it regularly loses whole windows full of tabs, even though it is set to restore my session on startup. And there's no way to manually save sessions. It's hopeless.

Re: Technical Details on the Recent Firefox Add-On Outage

#148

Post by drtillberg »

Someone1234 wrote: post #75
lllr_finger wrote: post #68

It might not have helped here, but updating certs every several years always leads to problems from my experience - people move on, processes get lost or outdated, etc. I'll accept the yearly annoyance to avoid those issues every time.

Agreed. My initial reaction when Let's Encrypt had you re-issue every 90 days was negative, but I was wrong. Very wrong. A 90 day re-issue forces you to have working re-issue infrastructure and procedures, and therefore you're less likely to get stung by an accidental expiration. Long expirations are a trap, a very easy trap to fall into.

Why not update long-duration keys every 90 days? That way you're never close to expiration, ever, best of both worlds.

Re: Technical Details on the Recent Firefox Add-On Outage

#149

Post by cortesoft »

StavrosK wrote: post #79
johncolanduoni wrote: post #32

Earlier quoted context omitted.

For the time stamp method to work, you need a trusted mechanism to attest that the timestamp is correct, otherwise the mechanism is useless (an attacker with an outdated private key can just backdate the timestamp in the executable and then sign it). Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same. I’m not saying they shouldn’t, but it is a significant piece of…

Can you explain why you need the current time anywhere in this? Say I download a ten-year-old addon, it's signed by a valid signature from that root, with a valid signing date. What's the problem? Are we worried that someone will steal an old/expired cert and have control over a user's clock?

Imagine you have an old expired key... you take your new malicious extension, and sign it with the expired key and a time stamp that says it was signed at a time the key was still valid.

Without some other verification mechanism, you can't tell the difference between this and an actual signature signed when the key WAS valid

Post reply on HN