Live data from Hacker News

O2 outage due to expired Ericsson certificate

ericsson.com

61–70 of 96 posts

Re: O2 outage due to expired Ericsson certificate

#61
post #46

Same risk with domains and DNS - when people question why CSCGlobal or UltraDNS charge so much compared to cheaper alts - it's because they have your back when you miss something

The only way Ultra have your back is ripping the shirt off you in overage charges

Re: O2 outage due to expired Ericsson certificate

#62
post #43

Earlier quoted context omitted.

Why is it better than GPG? (“Blockchain” is not a valid answer)

The idea is to use economic incentives to keep people honest rather than trusting centralized authorities which can be more easily compromised

GPG doesn't use a centralized authority. It uses a decentralized "Web Of Trust" model.

Re: O2 outage due to expired Ericsson certificate

#63

It took some warming to, but I have come round to appreciate letsencrypt's short certificate lifetimes. Monthly renewal should be maximum for any system, but ideally you'd want to go weekly. Assuming your renewal is automated, I don't see any downside, only benefits. It properly internalises cert renewal as part of standard system operations, bringing it into your daily ops instead of having it as some scary gray und…

Your proposal may lead to an unintended DDoS of the CA.

Re: O2 outage due to expired Ericsson certificate

#64
post #55

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…

I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.

Adding delays to responses can also work too.

Years ago, during the tail end of the Age of XML, I had a conversation with one of the admins over at W3C.

To make your app work you're supposed to cache all schemas locally but an inordinate amount of people don't. Their apps will be slow because of that, but work anyway. So tons of traffic goes to w3.org. They can't remove those schemas because then nobody can use them the right way (download them 1x and keep the result). But how do you get people's attention?

I pointed out to him that a human downloading the schema isn't going to notice if the request takes .3 seconds or 3 seconds, but someone loading that schema 100x in a loop is sure going to be incentivized to figure out where the extra 5 minutes of processing time came from. He allowed that I might be onto something there but I never heard back about whether he tried it.

You want to kill off an API, assign it resources (hardware, bandwidth) that get progressively smaller, or add artificial waits that get progressively bigger. Depending on how horizontally scalable your app is (is it easier to add a sleep or to pull servers out of the cluster?).

Re: O2 outage due to expired Ericsson certificate

#66
post #24

Earlier quoted context omitted.

One good reason is that if you buy a domain name that somebody else has used in the past, they don't have an infinite valid SSL certificate for your domain.

Would it not be possible to expire the cert if the domain expires?

No, that would be "revocation". Expiration is relatively easy to implement because the expiration date is known in advance and so you can simply put the expiration date in the certificate when it is issued. Revocation is relatively difficult because you need to continually check some database for revocation information — that's where CRLs, OCSP, and the like come in. And there's a lot of complexity under that hood, which, once the dust settles, boils down to just issuing very-short-lived certificates under a different guise.

Re: O2 outage due to expired Ericsson certificate

#67
post #55

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…

I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.

How long would it take before developers started wrapping their API and CDN calls in rapid-firing loops because of this?

Re: O2 outage due to expired Ericsson certificate

#68
post #55

Earlier quoted context omitted.

I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.

How long would it take before developers started wrapping their API and CDN calls in rapid-firing loops because of this?

It already exists and has borrowed the name "resilience engineering" from the construction and engineering fields. Netflix has some interesting blog posts on how they deal with transient faults and resilience in general. Implementing concepts like circuit breakers.

Have a search for libraries in your favorite language, I'm sure something will already exist. I've personally used Polly in .NET.

https://github.com/App-vNext/Polly

Re: O2 outage due to expired Ericsson certificate

#69

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…

Maybe it would be a good idea for people to put an expiration date check in whatever they use to monitor the rest of the machines. Or put the expiry date on a shared calendar.

We do both, and they start alerting two months in advance. This isn't rocket surgery, new, or much different than remembering people's birthdays.

But a surprising number of technical people are simply incapable of managing events farther out than next week, and a surprising number of their managers are similarly incapable of making sure someone treats it as important.

Makes me almost want to start a betting pool on who will be the next HugeCo to down themselves this way.

Edit: And let me join the chorus in saying deliberately inducing random errors as a way to draw attention is a terrible idea.

Post reply on HN