Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Technical Details on the Recent Firefox Add-On Outage

hacks.mozilla.org

Post reply on HN
Print view View original

71–80 of 279 posts

Re: Technical Details on the Recent Firefox Add-On Outage

#71

Post by gilrain »

callahad wrote: post #61
gilrain wrote: post #30

Earlier quoted context omitted.

As someone who opted out of Studies after the last abuse, I felt betrayed that my addons were, effectively, held hostage behind enabling both telemetry and Studies. I decided to wait. It’s bad optics at the very least. Users who opted in for the update were in fact entered into studies they explicitly wouldn’t have wanted to be in without the lure of an earlier update.

I'm sorry that you felt like you were held hostage by Telemetry/Studies. With the exception of the hotfix, we disabled rolling out new Studies during the incident, and will not be re-enabling them until some time after Monday next week. We are also completely deleting all Telemetry and Studies data received in the week following the incident to ensure we respect people who had concerns like yours, but enabled Studies…

That is excellent news!

Re: Technical Details on the Recent Firefox Add-On Outage

#72

Post by contravariant »

johncolanduoni wrote: post #32
jzl wrote: post #11

Looks like a good read. I haven't finished reading it yet, but there's something I still don't get ... Windows and macOS both have a signing infrastructure for apps. The rules of that infrastructure dictate only that apps must have been signed by a valid certificate at the time they were signed. That way old app downloads don't need to be periodically re-signed just to account for expiring certificates. I can downloa…

For the time stamp method to work, you need a trusted mechanism to attest that the timestamp is correct, otherwise the mechanism is useless (an attacker with an outdated private key can just backdate the timestamp in the executable and then sign it). Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same. I’m not saying they shouldn’t, but it is a significant piece of…

Maybe I'm jumping the gun here, but doesn't is seem that this kind of problem is exactly the kind of problem where a blockchain solution would be useful?

Or at least more useful than the 'when all you have is a hammer' style applications of blockchains.

Re: Technical Details on the Recent Firefox Add-On Outage

#74

Post by Someone1234 »

ghostly_s wrote: post #73

If anyone's wondering if this answers the Actual Question of why the cert was allowed to expired, don't waste your time, it doesn't. I guess implicitly that's a "social detail"?

What are you after?

This clearly says they screwed up and that internal procedures need to be changed so it cannot recur. If this was in fact an oversight/accident, what other kind of explanation can they give?

> This was due to an error on our end: we let one of the certificates used to sign add-ons expire

And:

> We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix.

Re: Technical Details on the Recent Firefox Add-On Outage

#75

Post by Someone1234 »

lllr_finger wrote: post #68

It might not have helped here, but updating certs every several years always leads to problems from my experience - people move on, processes get lost or outdated, etc. I'll accept the yearly annoyance to avoid those issues every time.

Agreed.

My initial reaction when Let's Encrypt had you re-issue every 90 days was negative, but I was wrong. Very wrong. A 90 day re-issue forces you to have working re-issue infrastructure and procedures, and therefore you're less likely to get stung by an accidental expiration.

Long expirations are a trap, a very easy trap to fall into.

Re: Technical Details on the Recent Firefox Add-On Outage

#76

Post by tialaramex »

ziddoap wrote: post #14
shawnz wrote: post #7

Earlier quoted context omitted.

It's clearly just a disclaimer to show his respectful intent, since the paragraphs following that are addressing failures of his colleagues.

Understood, and I explained a bit more of my thoughts in another comment. I understand I'm being a bit pedantic. But, as a user who is pretty concerned with a CA having a certificate issue of this scale, which affected this many people, I don't care about the authors respectful intent to their colleagues. Tell them in person, in an internal memo, whatever. It's simply not a lesson learned.

Mozilla, unlike all of the other major Trust Stores does not actually operate a publicly trusted CA.

Microsoft, Apple, and Google all do, the biggest Trust Store I can think of that does NOT operate a CA other than Mozilla is Oracle (who manage the default Java trust store)

This puts Mozilla in a good situation of independence IMNSHO.

At Google we know it takes actual effort to make sure that corporate firewalls protect the Trust Store function from being sabotaged by the convenience of, say, Google's Cloud function or people in their SRE wanting to keep the lights on for Chrome users. One day Google's CA people accidentally minted certs that Chrome wouldn't trust for Google web sites, and there had to be mechanisms in place to ensure that "Just tell the Chrome engineers to ignore it" wasn't a viable fix, because that would be _easier_ but it destroys any value in that trust store.

Re: Technical Details on the Recent Firefox Add-On Outage

#77

Post by preinheimer »

dave5104 wrote: post #70
steve19 wrote: post #66

Earlier quoted context omitted.

It was a couple of days before the Android fix was released. For many users this was a multiday affair. Some lost data (containers). You were lucky you extentions were not disabled before the fix, but for many people this was a major problem.

I unfortunately lost my container data (luckily I only have a couple set up). To get the fix though, I had to opt in to the Firefox studies. Apparently I had opted out at some point in the past.

Probably when they stealth installed an extension to promote Mr. Robot without telling you.

Re: Technical Details on the Recent Firefox Add-On Outage

#78

Post by simias »

contravariant wrote: post #72
johncolanduoni wrote: post #32

Earlier quoted context omitted.

For the time stamp method to work, you need a trusted mechanism to attest that the timestamp is correct, otherwise the mechanism is useless (an attacker with an outdated private key can just backdate the timestamp in the executable and then sign it). Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same. I’m not saying they shouldn’t, but it is a significant piece of…

Maybe I'm jumping the gun here, but doesn't is seem that this kind of problem is exactly the kind of problem where a blockchain solution would be useful? Or at least more useful than the 'when all you have is a hammer' style applications of blockchains.

Given that you have a clear authority here (Mozilla) I don't see why you'd query a blockchain instead of querying signatures.mozilla.org

As always Blockchains are a good solution only for a tiny group of problems.

Re: Technical Details on the Recent Firefox Add-On Outage

#79

Post by StavrosK »

johncolanduoni wrote: post #32
jzl wrote: post #11

Looks like a good read. I haven't finished reading it yet, but there's something I still don't get ... Windows and macOS both have a signing infrastructure for apps. The rules of that infrastructure dictate only that apps must have been signed by a valid certificate at the time they were signed. That way old app downloads don't need to be periodically re-signed just to account for expiring certificates. I can downloa…

For the time stamp method to work, you need a trusted mechanism to attest that the timestamp is correct, otherwise the mechanism is useless (an attacker with an outdated private key can just backdate the timestamp in the executable and then sign it). Windows code signing uses a server Microsoft runs to provide this, and Mozilla would need to do the same. I’m not saying they shouldn’t, but it is a significant piece of…

Can you explain why you need the current time anywhere in this? Say I download a ten-year-old addon, it's signed by a valid signature from that root, with a valid signing date. What's the problem?

Are we worried that someone will steal an old/expired cert and have control over a user's clock?

Post reply on HN