We use Erlang for a high availability of nine nines, so long as we remember to renew our certs.
[deleted]
O2 outage due to expired Ericsson certificate
31–40 of 96 posts
Re: O2 outage due to expired Ericsson certificate
#32Earlier quoted context omitted.
Yes. Even then, a few years is too long, because you forget about the need for renewal. I would say that the "proper" use of certificate expiry is the way LetsEncrypt and other ACME providers do it: it's set so low that you need an automated renewal process in order to make the certificate at-all useful.
But then you need to make sure that the automated cert renewal system is still working...
But when automated cert-renewal breaks, it immediately breaks, and you start getting "cert renewal failed" messages in your email prompting you to action—but at that point, you still have time remaining to fix it before the cert expires, since ACME impls tend not to renew the cert exactly one second before it expires, but rather more like a day or two before it would expire.
Unlike cert expiry, where the first you hear about it is when your production system stops working.
Re: O2 outage due to expired Ericsson certificate
#33Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one minute and 100% untrustworthy the next minute. Having the failure rate ramp up slowly would give advance warning before everything has gone completely pear-shaped.
In the case of Ericsson, this might have allowed an emergency certificate update before all the O2 systems could no longer be automatically updated. Once your network is completely down, bringing it back up remotely is hard.
Re: O2 outage due to expired Ericsson certificate
#34Ignoring letting it expire in the first place. The surprising part is it took over 24 hours to restore service. I currently still have 3G only, and that's struggling (apparently 4G will follow).
Still only 4G here too. Apparently we can expect it to return tomorrow morning. O2 were asked if we'll be compensated. They said they'll "apologise in an O2 way" but couldn't confirm what an "O2 way" is.
Apologise in an O2 way? Gosh are we back to BT era.
See what we can do. O2.
Re: O2 outage due to expired Ericsson certificate
#35Ooh - I was wondering what the hell was going on. I assumed I had dropped the darn thing one too many times. Glad to see the HN grapevine works on wifi. It is a reminder of just how fragile this digital world still is - we are taking technology designed to survive nuclear war, and adding single points of failure. Let's look at mesh networking again.
It wasn't just the phones or just data on phones. GWR ticket kiosks went offline. Bus stop digital timetables went offline. On the phone, SMS delievery went wonky i.e. same SMS being delivered more than once but on the sender's phone it showed up as not delivered !
Re: O2 outage due to expired Ericsson certificate
#36Ignoring letting it expire in the first place. The surprising part is it took over 24 hours to restore service. I currently still have 3G only, and that's struggling (apparently 4G will follow).
Still only 4G here too. Apparently we can expect it to return tomorrow morning. O2 were asked if we'll be compensated. They said they'll "apologise in an O2 way" but couldn't confirm what an "O2 way" is.
If O2 is like most phone companies, my reaction to this threat would be to beg them not to.
Re: O2 outage due to expired Ericsson certificate
#37Earlier quoted context omitted.
Yes. Even then, a few years is too long, because you forget about the need for renewal. I would say that the "proper" use of certificate expiry is the way LetsEncrypt and other ACME providers do it: it's set so low that you need an automated renewal process in order to make the certificate at-all useful.
But if it's automated, aren't we back to "forget about it for 25 years"?
[deleted]
Re: O2 outage due to expired Ericsson certificate
#38Ignoring letting it expire in the first place. The surprising part is it took over 24 hours to restore service. I currently still have 3G only, and that's struggling (apparently 4G will follow).
Still only 4G here too. Apparently we can expect it to return tomorrow morning. O2 were asked if we'll be compensated. They said they'll "apologise in an O2 way" but couldn't confirm what an "O2 way" is.
I assume that means they’ll apologise by ramping up your monthly rate.
Re: O2 outage due to expired Ericsson certificate
#39Why do certificates expire? How is it acceptable to have a piece of data somewhere that contains a timebomb that must be periodically defused? I do not see how this helps security, and it is particularly ridiculous that normal behavior is that one day the system works normally and is deemed secure, and the next day it is so insecure and dangerous that communication simply fails.
Cryptography is basically a computational treadmill: you want to make it cheap enough that it's not burdensome for the actual users, but that reversing the information without the key is computationally expensive. Processing power, especially for the highly parallelizable task of grinding through potential keys, follows an exponential curve; ergo, even the present exponential gap is not a long-term protection mechani…
I think the much more important reason for expiring certificates is to minimize the damage in the case that your private keys are stolen, which is far more likely than them being brute forced.
Re: O2 outage due to expired Ericsson certificate
#40If only the mechanisms that check certificates could provide warnings of impending expiry - 1,3,7,30 days would be prudent.
Though companies should be doing at least a yearly audit of certificates and calendering any that will need renewing. As I'm sure they do with domain names already.