Live data from Hacker News

Technical Details on the Recent Firefox Add-On Outage

hacks.mozilla.org

1–10 of 279 posts

Re: Technical Details on the Recent Firefox Add-On Outage

#2
>First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted.

It's a bit disheartening to see the "Lessons" section opened with this. I understand that everyone worked very hard to get things up and running - but that is not a lesson learned. That is self-back-patting.

Re: Technical Details on the Recent Firefox Add-On Outage

#3
post #2

>First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted. It's a bit disheartening to see the "Lessons" section opened with this. I understand that everyone worked very hard to get…

The post-mortem hasn't even started yet; now that 66.0.5 is out and the initial incident response is finally winding down, now they can move on to that.

Re: Technical Details on the Recent Firefox Add-On Outage

#4
post #3
post #2

>First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted. It's a bit disheartening to see the "Lessons" section opened with this. I understand that everyone worked very hard to get…

The post-mortem hasn't even started yet; now that 66.0.5 is out and the initial incident response is finally winding down, now they can move on to that.

I don't quite follow how this reply relates to the first paragraph under the "Lessons" section of the linked article.

Re: Technical Details on the Recent Firefox Add-On Outage

#5
post #2

>First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted. It's a bit disheartening to see the "Lessons" section opened with this. I understand that everyone worked very hard to get…

It's context for the next paragraph.

> With that said, obviously this isn’t an ideal situation and it shouldn’t have happened in the first place. We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix.

Re: Technical Details on the Recent Firefox Add-On Outage

#6
>An important feature here is that the new certificate has the same subject name and public key as the old certificate, so that its signature on the End-Entity certificate is valid.

Shouldn't it be impossible to generate a new cert (with a different expiry date) that ends up having the same public key as an existing cert?

Re: Technical Details on the Recent Firefox Add-On Outage

#7
post #2

>First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted. It's a bit disheartening to see the "Lessons" section opened with this. I understand that everyone worked very hard to get…

It's clearly just a disclaimer to show his respectful intent, since the paragraphs following that are addressing failures of his colleagues.

Re: Technical Details on the Recent Firefox Add-On Outage

#8
After the last PR disaster - the Mr Robot tie-in - one of the ways Mozilla tried to make it right was that they promised that the survey system would never again be used for something that was not an experiment.

https://blog.mozilla.org/firefox/retrospective-looking-glass...

> A SHIELD study must be designed to answer a specific question.

Why have they abused it again here to deploy a hot fix, breaking their promise and policy that they put in place last time they messed up?

Or am I ignorant or some part of the story or technical details.

Re: Technical Details on the Recent Firefox Add-On Outage

#9

>An important feature here is that the new certificate has the same subject name and public key as the old certificate, so that its signature on the End-Entity certificate is valid. Shouldn't it be impossible to generate a new cert (with a different expiry date) that ends up having the same public key as an existing cert?

No, one public/private key pair can be used to generate or sign as many certificates as you like, as often as you like.

Re: Technical Details on the Recent Firefox Add-On Outage

#10
post #5
post #2

>First, I want to say that the team here did amazing work: they built and shipped a fix in less than 12 hours from the initial report. As someone who sat in the meeting where it happened, I can say that people were working incredibly hard in a tough situation and that very little time was wasted. It's a bit disheartening to see the "Lessons" section opened with this. I understand that everyone worked very hard to get…

It's context for the next paragraph. > With that said, obviously this isn’t an ideal situation and it shouldn’t have happened in the first place. We clearly need to adjust our processes both to make this and similar incidents it less likely to happen and to make them easier to fix.

I did gather that much.

In my _personal opinion_, it would have been much more palatable to start the lessons learned section with

>We’ll be running a formal post-mortem next week and will publish the list of changes we intend to make, but in the meantime here are my initial thoughts about what we need to do. First, we should have a much better way of tracking

And shoe-horn the back-patting in earlier or later.

Perhaps I'm being pedantic. Fair enough. But for me, I would rather see this issue addressed in a way that the severity of the situation requires. Back-patting is not on the plate for me.

Post reply on HN