Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft failed to rotate certificates for winget CDN on time

cdn.winget.microsoft.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Microsoft failed to rotate certificates for winget CDN on time

#31

Post by linsomniac »

vbezhenar wrote: post #3

While I appreciate TLS, this thing with certificate expiration is one of the biggest sources of downtime IMO. Something should be done about it. May be throw error not permanently but in a some probabilistic way. Like if 1 year certificate expired, after 3 months 25% of connections would fail. It'll allow eventually to find out about problem but it'll allow for connections to somewhat work, with few retries here an t…

Making silent, intermittent failures for 3 months sounds so, so much worse than just having it 100% fail at expiration. What are we trying to fix here?

Re: Microsoft failed to rotate certificates for winget CDN on time

#32

Post by RamRodification »

derefr wrote: post #19
WirelessGigabit wrote: post #12

Earlier quoted context omitted.

I solve this by certificate expiration monitoring and renewing the certificate at the 60 day mark. The expiration warning is configured so that it starts to yell at me if it passes that timeframe. That gives me plenty of time to fix it IF it goes wrong.

In your case, "something breaks in your automation" might mean that, by the time the cert is (about to be) in need of renewal, the notifications you set up are now going to an email account that doesn't exist any more, because you left the department got re-orged and...

If "monitoring" is set up as "send email to specific personal mailbox" then things are gonna suck a lot.

Re: Microsoft failed to rotate certificates for winget CDN on time

#34

Post by physicles »

vbezhenar wrote: post #3

While I appreciate TLS, this thing with certificate expiration is one of the biggest sources of downtime IMO. Something should be done about it. May be throw error not permanently but in a some probabilistic way. Like if 1 year certificate expired, after 3 months 25% of connections would fail. It'll allow eventually to find out about problem but it'll allow for connections to somewhat work, with few retries here an t…

I'd take it as a canary for how much the site owner actually cares about ops and security. If you can't be bothered to take the day or so to set up certbot and monitoring for when your certs are ~15 days from expiration, then that's very likely not the only thing you've cut corners on.

Re: Microsoft failed to rotate certificates for winget CDN on time

#35

Post by linker3000 »

janoc wrote: post #28
Jamie9912 wrote: post #22

Aaaand it's still not fixed. I think this just goes to show how much red tape there is around processes at Microsoft.

You realize it is Sunday, right? And that probably a lot of relevant people are not in the office today, even if the techops grunts who are on call are scrambling.

Day of the week is irrelevant for an organisation that big. If you can't escalate a major incident like this to a designated person - even if you have to wake them up - your process and business continuity plans are seriously flawed.

Re: Microsoft failed to rotate certificates for winget CDN on time

#36

Post by Eduard »

josephg wrote: post #20
Uvix wrote: post #5

Earlier quoted context omitted.

The maximum expiration time is now down to 13 months, for certs that need to be valid in a browser. And if you want to cycle yours more frequently, you can. But there's enough places that can't set up automated processes that trying to make it 90 days for everyone would be a lot of pain and a lot of broken sites.

> But there's enough places that can't set up automated processes Why can't they be automated? And anyway, this is the exact problem that short expiration times avoid! Systems that aren't set up for automation, and rely on someone once a year remembering some creaky, error prone process to get a new cert. Much better to force short expiration times so manual cert renewal is a thing of the past.

> Why can't they be automated?

E.g. because of regulatory requirements, chain of responsibility, a paper has to be signed with a pen, etc.

Re: Microsoft failed to rotate certificates for winget CDN on time

#37

Post by Kwpolska »

voytec wrote: post #25
josephg wrote: post #20

Earlier quoted context omitted.

> But there's enough places that can't set up automated processes Why can't they be automated? And anyway, this is the exact problem that short expiration times avoid! Systems that aren't set up for automation, and rely on someone once a year remembering some creaky, error prone process to get a new cert. Much better to force short expiration times so manual cert renewal is a thing of the past.

DV is not the only kind of certificates validation. I don't want to have to go through the OV/EV validation process several times a year, nor to validate 4 certificate issuances a year in advance. But if I wanted to, I can do so even now without being forced - request new certificate during it's validity period, and revoke the former one.

DV is the only kind that actually matters. Browsers do not display EV certificates in the address bar anymore, the verified identity is hidden in a panel or sometimes even invisible. If you want to pay extra for snake oil, you get to enjoy all the pain in the process. See also: https://www.troyhunt.com/how-everything-were-told-about-webs...

Re: Microsoft failed to rotate certificates for winget CDN on time

#38

Post by Ekaros »

WirelessGigabit wrote: post #12
thayne wrote: post #9

Earlier quoted context omitted.

> And its long enough that each time, I (and everyone else) can justify not bothering to automate the process And even worse, if you do automate it there is a pretty good chance something changes and breaks your automation by the time it is needed. And that is assuming you actually tested the automation before your new cert is close to expiring.

I solve this by certificate expiration monitoring and renewing the certificate at the 60 day mark. The expiration warning is configured so that it starts to yell at me if it passes that timeframe. That gives me plenty of time to fix it IF it goes wrong.

And how do you ensure your monitoring keeps working?

Re: Microsoft failed to rotate certificates for winget CDN on time

#39

Post by orra »

Eduard wrote: post #36
josephg wrote: post #20

Earlier quoted context omitted.

> But there's enough places that can't set up automated processes Why can't they be automated? And anyway, this is the exact problem that short expiration times avoid! Systems that aren't set up for automation, and rely on someone once a year remembering some creaky, error prone process to get a new cert. Much better to force short expiration times so manual cert renewal is a thing of the past.

> Why can't they be automated? E.g. because of regulatory requirements, chain of responsibility, a paper has to be signed with a pen, etc.

Interesting, but that sounds like speculation. Do you have any examples of regulatory requirements, as opposed to voluntarily-broken internal processes?

Re: Microsoft failed to rotate certificates for winget CDN on time

#40

Post by zinekeller »

Looking at crt.sh (https://crt.sh/?q=cdn.winget.microsoft.com), it seems that the certificate is issued automatically anyways but for some reason the updated certificate is not applied correctly. A bad screwup really, but more of did someone forget to check their logs for deployment errors rather than the common case of someone forgetting to manually update the certificate.
Post reply on HN