Let's Encrypt is down

I'm running a local dnscache instance (djbdns), perhaps that has something to do with it?

I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem.

OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our engineers observed a rather odd, but standard, web server behavior. When a server receives a request with multiple slashes one after another they will collapse them into a single slash. This caused our OCSP responder to consider requests that had this unusual encoding quirk invalid and would respond to with a '400 Bad Request' response. The fix seemed quite simple: disable the slash collapsing behavior.

Unfortunately, stopping this behavior surfaced a more serious issue. The AIA extension that we include in certificates we issue contains a URI for our OCSP server. This URI contains a trailing slash. According to RFC 6960 Appendix 1 an OCSP request using the GET method is constructed as follows 'GET {url}/{url-encoding of base-64 encoding of the DER encoding of the OCSPRequest}' where the url 'may be derived from the value of the authority information access extension in the certificate being checked for revocation'. A number of user agents take this quite literally and will construct the URL without inspecting the contents of the AIA extension meaning that they ended up with a double slash between the host name and the base64 encoded OCSP request. Before we disabled slash collapsing this was fine as the web server was silently fixing this problem. Once we stopped collapsing slashes we started seeing problems.

From our OCSP server's perspective a majority of the OCSP requests we were receiving were prepended with a slash and we were unable to decode them so we'd respond with a '400 Bad Request' response and move on. This coincided with a large number of previously cached responses on our CDN expiring, causing us to start getting hit with a large number of requests. Because we were responding with '400 Bad Request' responses we were setting explicit no-cache headers which meant we had a near 0% cache (CDN) offload rate and were hit with the full brunt of our OCSP request load at our origin servers. This caused our whole infrastructure to get bogged down.

And then gain a reputation for being unreliable?

> Or if you want to make very sure you don't turn people away, be down for 4 hours every month for any cert that has been in Let's Encrypt for more than a month or two, so you don't turn away new users. Because if you have a problem with a brief outage, the problem is in the user code, not Let's Encrypt.

Most users don't care. If it's not working reliably for them, they'll just move to something that does. Maybe there is an issue in their code that should be addressed, but I seriously doubt they'd care to have that pointed out when they're suddenly offline.

Anyone doing proper testing of their software/infrastructure should have a testing environment anyway. I'd take your proposal and modify it to: Let's Encrypt should offer testing servers which are down for well defined periods throughout the day that people can use to test their platform against.

I think, in the past, mods have put an extra down-weight on "X is down" stories, once 'X' is back up.

Since this discussion now has interesting stuff related to Let's Encrypt—and products which use Let's Encrypt—I hope the mods are willing to forgo the down weight, and instead just change the post title to something like "Let's Encrypt Was Down".