Technical Details on the Recent Firefox Add-On Outage • Hacker News Board

Re: Technical Details on the Recent Firefox Add-On Outage

#191

Post by sohkamyung » Fri, May 10, 2019, 3:48 AM UTC

sohkamyung wrote: post #137

Related to this: Mozilla has deleted Telemetry data for those users who enabled Telemetry to get the hot-fix [1] [1] https://twitter.com/firefox/status/1126593558490693632

Edit: a error on my part. Mozilla is deleting ALL Telemetry data collected during that time period, not just from certain users. From the post linked from the tweet [1]

>In order to respect our users’ potential intentions as much as possible, based on our current set up, we will be deleting all of our source Telemetry and Studies data for our entire user population collected between 2019-05-04T11:00:00Z and 2019-05-11T11:00:00Z.

[1] https://blog.mozilla.org/blog/2019/05/09/what-we-do-when-thi...

Re: Technical Details on the Recent Firefox Add-On Outage

#192

Post by jrochkind1 » Fri, May 10, 2019, 3:55 AM UTC

Interestingly, it seems like it's not just that they didn't realize the certificate expire date was zooming past; they didn't even realize that particular certificate expiring was a thing. If they had noticed 6 months ago it was about to expire -- they still would have had to figure out what to do about it, there wasn't an actual documented procedure in place to swap in a new non-expired cert without disruption to FF users with existing signed add-ons installed. If they had known the expire date was upcoming, they just would have had more time to figure out what to do about it in a leisurely fashion.

I wonder if the engineers who implemented the original system had a clear idea of what would be done when the cert expire approached and it just never got documented (or FF accidentally changed to make it no longer applicable?), or if they just figured they'd figure that out years later when the date approached.

Re: Technical Details on the Recent Firefox Add-On Outage

#193

Post by ekianjo » Fri, May 10, 2019, 4:01 AM UTC

lifthrasiir wrote: post #146

ekianjo wrote: post #138

I love how they paint the picture that certificates "unfortunately expired" as if it were an act of god or something. Surely one cannot see it coming! No mention at all why nobody was checking certificates expiry.

The post seems to imply that it was a simple overlook (which is frequent when every such thing is not formally tracked). I agree that it is hard. > We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make, but in the meantime here are my initial thoughts about what we need to do. First, we should have a much better way of tracking the status of everything in Firefox that…

> a simple overlook

I'm sorry but when your business is to enable 30 000 extensions to work your business is also to check that certificates that enable such extensions don't fail tomorrow. That's the core of one's job, not even a side project or something.

Re: Technical Details on the Recent Firefox Add-On Outage

#194

Post by dwaite » Fri, May 10, 2019, 4:03 AM UTC

jzl wrote: post #11

Looks like a good read. I haven't finished reading it yet, but there's something I still don't get ... Windows and macOS both have a signing infrastructure for apps. The rules of that infrastructure dictate only that apps must have been signed by a valid certificate at the time they were signed. That way old app downloads don't need to be periodically re-signed just to account for expiring certificates. I can downloa…

An OS owns and can enforce policies around the filesystem. For example, a future release of macOS might say that files cannot be both executable and writable, and the only way a file can be marked executable is by the security system validating a signature, or an administrative user override.

Firefox has to deal with malware injecting extensions outside the normal browser process, so I'm not surprised they would default to having periodic re-checks.

Re: Technical Details on the Recent Firefox Add-On Outage

#195

Post by PudgePacket » Fri, May 10, 2019, 4:04 AM UTC

> Note that this is the same logic we use for validating TLS certificates, so it’s relatively well understood code that we were able to leverage.

Don't know whether to be happy or scared that the TLS validation code is "relatively well understood" :D ! I assume it's just a sub-optimal choice of phrasing.

Re: Technical Details on the Recent Firefox Add-On Outage

#196

Post by AnaniasAnanas » Fri, May 10, 2019, 4:12 AM UTC

nocman wrote: post #99

pbhjpbhj wrote: post #91

Earlier quoted context omitted.

So that means going forward if Mozilla don't want you to have an add-on then you won't be able to enable it?

Unless you run an unbranded builds ( https://wiki.mozilla.org/Add-ons/Extension_Signing#Unbranded... ), or the Developer Edition ( https://www.mozilla.org/en-US/firefox/developer/ ), where setting xpinstall.signatures.required to false still works, then yes, Mozilla can prevent you from using an add-on. More than "going forward", this has been the case for a while now. It's been long enough ago that they disabled tha…

Sadly the unbranded builds do not have updates enabled and the developer edition is basically beta.

Re: Technical Details on the Recent Firefox Add-On Outage

#197

Post by Twirrim » Fri, May 10, 2019, 4:15 AM UTC

sjwright wrote: post #188

Twirrim wrote: post #90

Earlier quoted context omitted.

I find it hard to marry up "Firefox user that cares about privacy" with being happy about the study mechanism. Don't you, from a privacy perspective, find it more than a little disturbing that the study mechanism has so much access to internal APIs in Firefox that it can install certificates without your involvement? That seems like a crazy security risk, let alone privacy risk. It's built in and enabled _by default_…

It’s funny how the mind works. We tend to accept the far greater risk of allowing automated software updates (either in-app or blindly trusting your package manager e.g. apt-get upgrade ) but the Firefox Studies mechanism which does a minuscule, strict subset of what any automatic update can do is somehow a “crazy security risk”.

In the case of apt-get, there's a release process with signed packages and open source code that the distributions adhere to, publicly visible oversight etc. End users specifically make the choice, either to run the upgrade process themselves, or set up automated upgrades on the understanding of what processes things have gone through, and the ability to verify.

The Studies mechanism occurs silently, running private code, without specific user action, and with "extensive access to Firefox internal APIs".

Re: Technical Details on the Recent Firefox Add-On Outage

#198

Post by AnaniasAnanas » Fri, May 10, 2019, 4:20 AM UTC

sjwright wrote: post #188

Twirrim wrote: post #90

Earlier quoted context omitted.

I find it hard to marry up "Firefox user that cares about privacy" with being happy about the study mechanism. Don't you, from a privacy perspective, find it more than a little disturbing that the study mechanism has so much access to internal APIs in Firefox that it can install certificates without your involvement? That seems like a crazy security risk, let alone privacy risk. It's built in and enabled _by default_…

It’s funny how the mind works. We tend to accept the far greater risk of allowing automated software updates (either in-app or blindly trusting your package manager e.g. apt-get upgrade ) but the Firefox Studies mechanism which does a minuscule, strict subset of what any automatic update can do is somehow a “crazy security risk”.

Normal update channels usually ask me (or let me set it so that they have to ask me) if I want to apply the update and I make sure to at least read the changelog if not skim through the source. In addition to that I trust the Debian team much more than I trust the Mozilla team. The studies in contrast are both for anti-user things such as telemetry and the mr robot thing, are inconvenient to disable, are not transparent (it would be difficult for me to find the source of a specific study), happen silently, and skip the Debian team's judgement.

Re: Technical Details on the Recent Firefox Add-On Outage

#199

Post by lifthrasiir » Fri, May 10, 2019, 4:24 AM UTC

ekianjo wrote: post #193

lifthrasiir wrote: post #146

Earlier quoted context omitted.

The post seems to imply that it was a simple overlook (which is frequent when every such thing is not formally tracked). I agree that it is hard. > We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make, but in the meantime here are my initial thoughts about what we need to do. First, we should have a much better way of tracking the status of everything in Firefox that…

> a simple overlook I'm sorry but when your business is to enable 30 000 extensions to work your business is also to check that certificates that enable such extensions don't fail tomorrow. That's the core of one's job, not even a side project or something.

I don't claim to know what actually happened, but one possible cause is that the HSM (required for issuing and renewing intermediate certificates, mentioned in the post) might require a right person and right schedule to operate. The use of an HSM means that you don't normally use it to issue certificates and you should only touch it a few times a year. As a result there may have been only a few people with an effective knowledge of the HSM; when they can't be scheduled for some various reasons, well, it may be forgotten. While it of course sucks, something like this happens anywhere anytime.

Re: Technical Details on the Recent Firefox Add-On Outage

#200

Post by BuckRogers » Fri, May 10, 2019, 4:35 AM UTC

preinheimer wrote: post #77

dave5104 wrote: post #70

Earlier quoted context omitted.

I unfortunately lost my container data (luckily I only have a couple set up). To get the fix though, I had to opt in to the Firefox studies. Apparently I had opted out at some point in the past.

Probably when they stealth installed an extension to promote Mr. Robot without telling you.

I caught that before it was posted in the news, and it really disturbed me to see it. "MY REALITY IS DIFFERENT THAN YOURS" or whatever it said. What's that about, talk about unprofessional. That and removing Live Bookmarks really degraded my trust, this weird Pocket thing (seems like a feature, not a product, and one that should've been built from scratch to maximize uniqueness), and I've been using Firefox and only Firefox since 2002. They're definitely losing me over time, it's the community that saves Firefox. Mozilla enables that but shouldn't really get credit for it.

Currently testing out "Edgium" though, and I can already say it's the one browser that could pull me away.