Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

O2 outage due to expired Ericsson certificate

ericsson.com

Post reply on HN
Print view View original

71–80 of 96 posts

Re: O2 outage due to expired Ericsson certificate

#71

Post by slimsag »

_jal wrote: post #69
mhandley wrote: post #33

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…

Maybe it would be a good idea for people to put an expiration date check in whatever they use to monitor the rest of the machines. Or put the expiry date on a shared calendar. We do both, and they start alerting two months in advance. This isn't rocket surgery, new, or much different than remembering people's birthdays. But a surprising number of technical people are simply incapable of managing events farther out th…

> whatever they use to monitor the rest of the machines.

"Ah, are we supposed to do that too?"

Re: O2 outage due to expired Ericsson certificate

#72

Post by marbletiles »

walshemj wrote: post #53
Bokanovsky wrote: post #5

This was a major outage in the UK causing millions of people not to have data access on their phones. https://www.bbc.co.uk/news/business-46464730

And some major major emergency ass pulling I was told by insiders.

"Ass pulling"?

Re: O2 outage due to expired Ericsson certificate

#73

Post by megy »

mhandley wrote: post #33

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…

Are you insane? That would be fun to diagnose.

Re: O2 outage due to expired Ericsson certificate

#74

Post by jjeaff »

tqkxzugoaupvwqr wrote: post #63
nothrabannosir wrote: post #57

It took some warming to, but I have come round to appreciate letsencrypt's short certificate lifetimes. Monthly renewal should be maximum for any system, but ideally you'd want to go weekly. Assuming your renewal is automated, I don't see any downside, only benefits. It properly internalises cert renewal as part of standard system operations, bringing it into your daily ops instead of having it as some scary gray und…

Your proposal may lead to an unintended DDoS of the CA.

Let's Encrypt recommends a daily cron job for cert renewal. The certbot only actually pings the servers for a renewed cert if the current cert is within 30 days of expiration.

Re: O2 outage due to expired Ericsson certificate

#75

Post by Crosseye_Jack »

Bokanovsky wrote: post #5

This was a major outage in the UK causing millions of people not to have data access on their phones. https://www.bbc.co.uk/news/business-46464730

Not just data but also calls and text. Mother is on o2 and until about 6pm I was unable to call, was able to text her from about 4pm. Between 4 and 6 the calls were just failing with “called failed” and not even going to Voicemail, but texts was go though and I would even get delivery reports, just that calls wouldn’t connect.

Re: O2 outage due to expired Ericsson certificate

#76

Post by jcrawfordor »

jtl999 wrote: post #20
stdplaceholder wrote: post #18

Earlier quoted context omitted.

I know it's happened to every company where I've worked. It happens so rarely, though, that people don't have enough opportunity to learn from it. Even at Google they were on their Nth such outage for a large value of N before it became apparent that no certificate should ever expire at 23:59:59 on December 31, or otherwise outside of normal operating hours. Seriously 20 years of organizational knowledge required to…

Instead of waiting last minute, you'd think a large company would have planning to renew certificates X amount of time before they expire. Alas I understand it's not that simple.

In my small organization we had planning and multiple reminders to renew the cert well before it expired, and we did. Due to a miscommunication between myself and a coworker, the new cert sat ready for nearly two months without ever being added to the configuration (we were both certain the other had done it, naturally).

There's a remarkable number of ways for this simple thing to go wrong. To prevent a future repeat, we got rid of our calendar reminders (which we started ignoring once we both thought the change had been made) and wrote a script that emailed us based on the time to expiration of the live cert. This is a much better method.

Of course, give us enough years and I'm sure we'll manage to find a way to get this new setup wrong.

Re: O2 outage due to expired Ericsson certificate

#77

Post by closeparen »

mhandley wrote: post #33

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…

At this point I wouldn’t even contemplate using X.509 without a fully automated PKI issuing ephemeral certs. Paging humans is not the answer.

Re: O2 outage due to expired Ericsson certificate

#78

Post by umichguy »

jwdunne wrote: post #30
NeedMoreTea wrote: post #3

Ignoring letting it expire in the first place. The surprising part is it took over 24 hours to restore service. I currently still have 3G only, and that's struggling (apparently 4G will follow).

Still only 4G here too. Apparently we can expect it to return tomorrow morning. O2 were asked if we'll be compensated. They said they'll "apologise in an O2 way" but couldn't confirm what an "O2 way" is.

They are a bunch of O2 thieves.

Re: O2 outage due to expired Ericsson certificate

#79

Post by drannex »

megy wrote: post #73
mhandley wrote: post #33

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…

Are you insane? That would be fun to diagnose.

While I can totally see why this would be annoying to diagnose, throwing a specific form of a certificate expired error code could mitigate this issue.
Post reply on HN