[deleted]
O2 outage due to expired Ericsson certificate
81–90 of 96 posts
Re: O2 outage due to expired Ericsson certificate
#82Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…
No, this is not a good idea. Failures which are random are harder to diagnose. Something which works or not is much quicker to track down. The simple fact is that this sort of problem should've been dealt wiht a lot sooner, and a failure to do so is sheer incompetence. Where that incompetence lies, is up for debate. But given how these things usually run, it lies some levels above the people who didn't have the time…
Of course certificates should be managed properly, so this sort of thing should never happen. My suggestion is not instead of that, but in addition. The question is when your certificate management process has failed, what then? It's defense in depth.
In this case the certs expiry seems to have rendered it hard or impossible to remotely resurrect the systems. That's a very brutal failure when your whole network goes down in one go. Having random failures, with alarms going off before everything is down, is definitely better that having alarms going off with the entire network already down.
Re: O2 outage due to expired Ericsson certificate
#83If an ssl cert is expired on a website, you don't have access to the secured website.
In this case though, how does it affect Ericssons telcom equipment (core routers and switchs for mobileand voice) ?
Re: O2 outage due to expired Ericsson certificate
#84This was a major outage in the UK causing millions of people not to have data access on their phones. https://www.bbc.co.uk/news/business-46464730
Not just data but also calls and text. Mother is on o2 and until about 6pm I was unable to call, was able to text her from about 4pm. Between 4 and 6 the calls were just failing with “called failed” and not even going to Voicemail, but texts was go though and I would even get delivery reports, just that calls wouldn’t connect.
Yeah, O2 kept saying that calls and texts were not affected, but that was utter nonsense. I couldn't place a call most of the day, and texts would go out, but my phone would insist that sending failed. And then there was just no signal at all most of the afternoon.
Re: O2 outage due to expired Ericsson certificate
#85Earlier quoted context omitted.
And some major major emergency ass pulling I was told by insiders.
"Ass pulling"?
An Ass Pull is a moment when the writers pull something out of thin air in a less-than-graceful narrative development, violating the Law of Conservation of Detail by dropping a Plot-critical detail in the middle, or near the end of their narrative without Foreshadowing or dropping a Chekhov's Gun earlier on.
Re: O2 outage due to expired Ericsson certificate
#86It took some warming to, but I have come round to appreciate letsencrypt's short certificate lifetimes. Monthly renewal should be maximum for any system, but ideally you'd want to go weekly. Assuming your renewal is automated, I don't see any downside, only benefits. It properly internalises cert renewal as part of standard system operations, bringing it into your daily ops instead of having it as some scary gray und…
There is no need for enforced expiry of certificates, especially here where there are used internally on a controlled network.
If there is a requirement or good practice to periodically replace key-pairs then it should be a network management operation. Equipment should never stop working because it decided that a key-pair had expired.
X509 certificates are not technically optimal in many situations and, let's face it, the main driver for expiry dates is that certificate vendors want to sell more certificates...
Re: O2 outage due to expired Ericsson certificate
#87If only the mechanisms that check certificates could provide warnings of impending expiry - 1,3,7,30 days would be prudent. Though companies should be doing at least a yearly audit of certificates and calendering any that will need renewing. As I'm sure they do with domain names already.
Presumably this certificate was protecting some machine-to-machine connection. Then, no human would ever have seen those warnings.
Re: O2 outage due to expired Ericsson certificate
#88Certificates can be hard to manage across enterprises. I have a project coming across my desk next year specifically to manage expiring certs and track on going changes. The company has 20,000+ certs to manage for us and our customers.
I'd find a HN post about this project and it's results very interesting
There was an interesting post a few weeks back on Autotrader moving all of their online properties over to LetsEncrypt https://news.ycombinator.com/item?id=17949741
Re: O2 outage due to expired Ericsson certificate
#89Why do certificates expire? How is it acceptable to have a piece of data somewhere that contains a timebomb that must be periodically defused? I do not see how this helps security, and it is particularly ridiculous that normal behavior is that one day the system works normally and is deemed secure, and the next day it is so insecure and dangerous that communication simply fails.
Attestation of identity should be seen as a service, not a product. Revocation exists. The fact that certificates are pieces of data is a compromise, not an ideal design.
Most CAs already support OCSP, which is effectively one-day-long certificates: the client can contact the OCSP server for a signed response saying "yes, it's still unrevoked", or the server can include ("staple") such a response along with its certs. Some certs have the MustStaple extension, indicating that they should not be treated as valid unless a recent OCSP response is stapled to it.
If we had the computational resources to just not have certificates at all and have every client check the CA's current belief that a public key belongs to a name at each use (and magically avoid the associated privacy problems), that would be ideal. Certificates are an approximation.
One neat thing about 3-month certificates is that it's a meaningfully different human lifescale from a year (or multiple years): operators may not still be around in a year but will certainly be around in 3 months. Operators may just plan to manually renew in a few years, but generally decide they need to automate the process if it's every 3 months. (For extremely boring reasons, I manually update the certificate on my personal website every 3 months and it's a pain, and I think I am one of very few people who do manual updates to their Let's Encrypt certs.) So it forces people to think of certificates as running their end of a service with ongoing operational work, not a one-time transfer of data.
Re: O2 outage due to expired Ericsson certificate
#90It took some warming to, but I have come round to appreciate letsencrypt's short certificate lifetimes. Monthly renewal should be maximum for any system, but ideally you'd want to go weekly. Assuming your renewal is automated, I don't see any downside, only benefits. It properly internalises cert renewal as part of standard system operations, bringing it into your daily ops instead of having it as some scary gray und…
Your proposal may lead to an unintended DDoS of the CA.
Most CAs already operate an OCSP responder which supports querying daily (if not more frequently) to make sure a certificate still hasn't expired; it checks the database and responds with a signed and timestamped acknowledgement. (Those that don't need to support frequent queries of a static certificate revocation list file.) So this is already a level of requests that CAs have to be built for.
And even if it didn't, scaling certificates from once a year to once a week requires merely 50x capacity - a finite number, and something that's very easy to plan out. If CAs in the '90s and '00s were able to handle a certain once-a-year request load, surely CAs today can handle 50x that.