Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

O2 outage due to expired Ericsson certificate

ericsson.com

Post reply on HN
Print view View original

41–50 of 96 posts

Re: O2 outage due to expired Ericsson certificate

#41

Post by wcoenen »

derefr wrote: post #32
zimbatm wrote: post #25

Earlier quoted context omitted.

But then you need to make sure that the automated cert renewal system is still working...

But when automated cert-renewal breaks, it immediately breaks, and you start getting "cert renewal failed" messages in your email prompting you to action—but at that point, you still have time remaining to fix it before the cert expires, since ACME impls tend not to renew the cert exactly one second before it expires, but rather more like a day or two before it would expire. Unlike cert expiry, where the first you he…

But of course, the failure emails will go to an email address that was deactivated a couple years ago because an employee left/everybody fired in reorg/unused address cleanup/domain migration/technical error/...

Just this week I had someone complain to me that he was no longer getting build failure emails, and it turned out IT had disappeared our old @ addresses. That reminds me about an IT ticket I need to write.

Re: O2 outage due to expired Ericsson certificate

#43

Post by seibelj »

aaaaaaaaaab wrote: post #29
seibelj wrote: post #22

An interesting new project is Handshake which is attempting to use decentralization to remove centralized certificate authorities. Maybe it will help stop these and similar situations in the future. https://handshake.org/

Why is it better than GPG? (“Blockchain” is not a valid answer)

The idea is to use economic incentives to keep people honest rather than trusting centralized authorities which can be more easily compromised

Re: O2 outage due to expired Ericsson certificate

#44

Post by toast0 »

AltF4me wrote: post #24
jstanley wrote: post #17

Earlier quoted context omitted.

One good reason is that if you buy a domain name that somebody else has used in the past, they don't have an infinite valid SSL certificate for your domain.

Would it not be possible to expire the cert if the domain expires?

No. The certificate's expiration is fixed at time of issuance. You could set the expiration of the certificate to the expiration date of the domain, but the domain could be transferred, cancelled, or revoked before the expiration.

Re: O2 outage due to expired Ericsson certificate

#45

Post by [deleted] »

mhandley wrote: post #33

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…

[deleted]

Re: O2 outage due to expired Ericsson certificate

#47

Post by msbarnett »

RowanH wrote: post #13

Would be interesting to know how many people who've managed footprints for a reasonable period of time (say 5-10years) who haven't had a cert expire on them. Wouldn't be surprised if it's single digit %ages. So many human & tech error factors lead to this occurring and they're all the same old things. Staffing changes, spam filters, ignored warnings, skipped emails...

A really easy way to avoid this in any environment with Continuous Integration style tests running on everyone's work:

Add a test that just unconditionally fails on a certain date, like a week before your cert expires. Don't let any code review sign off on a merge of a fix the test until the new cert is in prod. Don't let any code promote between environments while tests are broken.

The problem with emails and warnings is they're all ignorable and therefore completely unsuitable for managing something as critical as a cert.

The secret is to create a straight-up error that absolutely interrupts every engineer in the organization's day until the cert is renewed. A development-halting error a week before cert expiration is a hell of a lot better than a business-halting error when it expires in prod.

Re: O2 outage due to expired Ericsson certificate

#50

Post by karambahh »

wil421 wrote: post #42

Certificates can be hard to manage across enterprises. I have a project coming across my desk next year specifically to manage expiring certs and track on going changes. The company has 20,000+ certs to manage for us and our customers.

I can't tell if you're sitting on a gold mine or about to get very, very depressed very, very quickly.

I am, however, pretty sure there's no middle ground.

Wish you all the best but I would never trade my job for yours, I value my sanity way more than that.

Post reply on HN