Same risk with domains and DNS - when people question why CSCGlobal or UltraDNS charge so much compared to cheaper alts - it's because they have your back when you miss something
O2 outage due to expired Ericsson certificate
61–70 of 96 posts
Re: O2 outage due to expired Ericsson certificate
#62Earlier quoted context omitted.
Why is it better than GPG? (“Blockchain” is not a valid answer)
The idea is to use economic incentives to keep people honest rather than trusting centralized authorities which can be more easily compromised
Re: O2 outage due to expired Ericsson certificate
#63It took some warming to, but I have come round to appreciate letsencrypt's short certificate lifetimes. Monthly renewal should be maximum for any system, but ideally you'd want to go weekly. Assuming your renewal is automated, I don't see any downside, only benefits. It properly internalises cert renewal as part of standard system operations, bringing it into your daily ops instead of having it as some scary gray und…
Re: O2 outage due to expired Ericsson certificate
#64Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…
I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.
Years ago, during the tail end of the Age of XML, I had a conversation with one of the admins over at W3C.
To make your app work you're supposed to cache all schemas locally but an inordinate amount of people don't. Their apps will be slow because of that, but work anyway. So tons of traffic goes to w3.org. They can't remove those schemas because then nobody can use them the right way (download them 1x and keep the result). But how do you get people's attention?
I pointed out to him that a human downloading the schema isn't going to notice if the request takes .3 seconds or 3 seconds, but someone loading that schema 100x in a loop is sure going to be incentivized to figure out where the extra 5 minutes of processing time came from. He allowed that I might be onto something there but I never heard back about whether he tried it.
You want to kill off an API, assign it resources (hardware, bandwidth) that get progressively smaller, or add artificial waits that get progressively bigger. Depending on how horizontally scalable your app is (is it easier to add a sleep or to pull servers out of the cluster?).
Re: O2 outage due to expired Ericsson certificate
#65Re: O2 outage due to expired Ericsson certificate
#66Earlier quoted context omitted.
One good reason is that if you buy a domain name that somebody else has used in the past, they don't have an infinite valid SSL certificate for your domain.
Would it not be possible to expire the cert if the domain expires?
Re: O2 outage due to expired Ericsson certificate
#67Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…
I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.
Re: O2 outage due to expired Ericsson certificate
#68Earlier quoted context omitted.
I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.
How long would it take before developers started wrapping their API and CDN calls in rapid-firing loops because of this?
Have a search for libraries in your favorite language, I'm sure something will already exist. I've personally used Polly in .NET.
Re: O2 outage due to expired Ericsson certificate
#69Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…
We do both, and they start alerting two months in advance. This isn't rocket surgery, new, or much different than remembering people's birthdays.
But a surprising number of technical people are simply incapable of managing events farther out than next week, and a surprising number of their managers are similarly incapable of making sure someone treats it as important.
Makes me almost want to start a betting pool on who will be the next HugeCo to down themselves this way.
Edit: And let me join the chorus in saying deliberately inducing random errors as a way to draw attention is a terrible idea.