O2 outage due to expired Ericsson certificate • Hacker News Board

Re: O2 outage due to expired Ericsson certificate

#21

Post by stdplaceholder » Thu, Dec 06, 2018, 10:48 PM UTC

jtl999 wrote: post #20

stdplaceholder wrote: post #18

Earlier quoted context omitted.

I know it's happened to every company where I've worked. It happens so rarely, though, that people don't have enough opportunity to learn from it. Even at Google they were on their Nth such outage for a large value of N before it became apparent that no certificate should ever expire at 23:59:59 on December 31, or otherwise outside of normal operating hours. Seriously 20 years of organizational knowledge required to…

Instead of waiting last minute, you'd think a large company would have planning to renew certificates X amount of time before they expire. Alas I understand it's not that simple.

You'd certainly think so. If you have frontend probers that exercise your accessible endpoints (HTTP or whatever) then those probes should fail when the certificate expires in less than 30 days. I couldn't comment on whether an organization like Ericsson or O2 would be expected to have such probers.

Re: O2 outage due to expired Ericsson certificate

#22

Post by seibelj » Thu, Dec 06, 2018, 10:50 PM UTC

An interesting new project is Handshake which is attempting to use decentralization to remove centralized certificate authorities. Maybe it will help stop these and similar situations in the future.

https://handshake.org/

Re: O2 outage due to expired Ericsson certificate

#23

Post by nemothekid » Thu, Dec 06, 2018, 10:51 PM UTC

admax88q wrote: post #15

modernpacifist wrote: post #10

Earlier quoted context omitted.

Seriously? Any half decent administrator knows that for every SSL certificate you install, you set a date some time in the future where you need to change it out for a new one. To your comment of "one day the system works normally and is deemed secure, and the next day it is so insecure and dangerous", this is working as intended. The certificate is to establish trust and identity along with encrypting the data in tr…

Expiry dates are in years, if a key is compromised, then an adversary has _years_ to exploit a MITM. However we already mitigate this with revocation lists. But if we can revoke certificates why do we have expiration dates? Seems to me expiration dates are rent seeking behaviour by certificate vendors.

This is a reason to shorten expiration times, not remove them (which companies like LetsEncrypt are doing)

Re: O2 outage due to expired Ericsson certificate

#24

Post by AltF4me » Thu, Dec 06, 2018, 10:51 PM UTC

jstanley wrote: post #17

admax88q wrote: post #15

Earlier quoted context omitted.

Expiry dates are in years, if a key is compromised, then an adversary has _years_ to exploit a MITM. However we already mitigate this with revocation lists. But if we can revoke certificates why do we have expiration dates? Seems to me expiration dates are rent seeking behaviour by certificate vendors.

One good reason is that if you buy a domain name that somebody else has used in the past, they don't have an infinite valid SSL certificate for your domain.

Would it not be possible to expire the cert if the domain expires?

Re: O2 outage due to expired Ericsson certificate

#25

Post by zimbatm » Thu, Dec 06, 2018, 10:53 PM UTC

derefr wrote: post #7

jstanley wrote: post #4

Earlier quoted context omitted.

I agree with you, but I suspect the intention is to prevent another type of timebomb. One where one day the system works and everything is considered secure, and then 25 years later nobody has given it a second thought.

Yes. Even then, a few years is too long, because you forget about the need for renewal. I would say that the "proper" use of certificate expiry is the way LetsEncrypt and other ACME providers do it: it's set so low that you need an automated renewal process in order to make the certificate at-all useful.

But then you need to make sure that the automated cert renewal system is still working...

Re: O2 outage due to expired Ericsson certificate

#26

Post by nickdothutton » Thu, Dec 06, 2018, 10:53 PM UTC

Reminds me of the 1990 network-wide outage AT&T suffered, aka "the big one of nine-oh". Caused by a bad update which took a couple of weeks to eventually bork the entire network.

Re: O2 outage due to expired Ericsson certificate

#27

Post by cnorthwood » Thu, Dec 06, 2018, 10:54 PM UTC

NeedMoreTea wrote: post #3

Ignoring letting it expire in the first place. The surprising part is it took over 24 hours to restore service. I currently still have 3G only, and that's struggling (apparently 4G will follow).

I suspect it was a root CA (they possibly had a private PKI) that expired - I saw a tweet suggesting that they had to manually apply a fix to every deployment of the software, which is consistent with having to change the trust chain.

Re: O2 outage due to expired Ericsson certificate

#28

Post by LinuxBender » Thu, Dec 06, 2018, 10:54 PM UTC

admax88q wrote: post #15

modernpacifist wrote: post #10

Earlier quoted context omitted.

Seriously? Any half decent administrator knows that for every SSL certificate you install, you set a date some time in the future where you need to change it out for a new one. To your comment of "one day the system works normally and is deemed secure, and the next day it is so insecure and dangerous", this is working as intended. The certificate is to establish trust and identity along with encrypting the data in tr…

Expiry dates are in years, if a key is compromised, then an adversary has _years_ to exploit a MITM. However we already mitigate this with revocation lists. But if we can revoke certificates why do we have expiration dates? Seems to me expiration dates are rent seeking behaviour by certificate vendors.

Just a few years ago, almost nothing checked the revocation lists. I revoked certs for some popular domains and was concerned about ssl caches and proxies... turns out, an owl heard it. An odd dog barked. No impact. Not even from the folks that embedded our certs onto their servers for legacy code reasons. Perhaps this has changed over the last couple of years.

Re: O2 outage due to expired Ericsson certificate

#29

Post by aaaaaaaaaab » Thu, Dec 06, 2018, 10:58 PM UTC

seibelj wrote: post #22

An interesting new project is Handshake which is attempting to use decentralization to remove centralized certificate authorities. Maybe it will help stop these and similar situations in the future. https://handshake.org/

Why is it better than GPG?

(“Blockchain” is not a valid answer)

Re: O2 outage due to expired Ericsson certificate

#30

Post by jwdunne » Thu, Dec 06, 2018, 11:00 PM UTC

NeedMoreTea wrote: post #3

Ignoring letting it expire in the first place. The surprising part is it took over 24 hours to restore service. I currently still have 3G only, and that's struggling (apparently 4G will follow).

Still only 4G here too. Apparently we can expect it to return tomorrow morning. O2 were asked if we'll be compensated. They said they'll "apologise in an O2 way" but couldn't confirm what an "O2 way" is.