51–60 of 96 posts
Ignoring letting it expire in the first place. The surprising part is it took over 24 hours to restore service. I currently still have 3G only, and that's struggling (apparently 4G will follow).
Still only 4G here too. Apparently we can expect it to return tomorrow morning. O2 were asked if we'll be compensated. They said they'll "apologise in an O2 way" but couldn't confirm what an "O2 way" is.
What is O2? It's not mentioned in the article but it's in the title.
This was a major outage in the UK causing millions of people not to have data access on their phones. https://www.bbc.co.uk/news/business-46464730
Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…
Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…
Certificates can be hard to manage across enterprises. I have a project coming across my desk next year specifically to manage expiring certs and track on going changes. The company has 20,000+ certs to manage for us and our customers.
Much like writing tests, it initially hurt my ego a bit, but I've come to like and proselytise it.
Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…
This is immediately results in the “it took us more than 24 hours to work out everything was failing, it needs to be 72 hours”. The best thing about that is that it results in a self extending policy - the longer it takes to age out an expired cert, the longer it takes for the failure volume to become noticeable, and then the longer to work out that the problem is cert expiry, and so the more likely they’ll need an extension (again).
The correct solution for this problem is for people to correctly manage their certs, which they should be doing anyway because the private keys are sufficiently important you should know exactly which are in use and where they are in use.
Would be interesting to know how many people who've managed footprints for a reasonable period of time (say 5-10years) who haven't had a cert expire on them. Wouldn't be surprised if it's single digit %ages. So many human & tech error factors lead to this occurring and they're all the same old things. Staffing changes, spam filters, ignored warnings, skipped emails...
A really easy way to avoid this in any environment with Continuous Integration style tests running on everyone's work: Add a test that just unconditionally fails on a certain date, like a week before your cert expires. Don't let any code review sign off on a merge of a fix the test until the new cert is in prod. Don't let any code promote between environments while tests are broken. The problem with emails and warnin…
Or just write a test that checks the date on the cert and conditionally fails.
If it's within 4 weeks, send email to x,y,z.
If it's within 2 weeks, send it to VP of x, y, z.
combination of failing test and email should lessen chance of it going unnoticed(email could possibly fail for whatever reason).
Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one m…
The simple fact is that this sort of problem should've been dealt wiht a lot sooner, and a failure to do so is sheer incompetence.
Where that incompetence lies, is up for debate. But given how these things usually run, it lies some levels above the people who didn't have the time to properly follow a poorly-described process in the face of demands to do other stuff instead.