Live data from Hacker News

Let's Encrypt is down

letsencrypt.status.io

141–150 of 167 posts

Re: Let's Encrypt is down

#141
post #79

Earlier quoted context omitted.

The simple config syntax and sane defaults is nice in Caddy. For example, 3 lines of config nets a https server with http/2 support and an A rating with Qualsys for ssl setup.

> sane defaults Failure to start if a CA is down is a "sane default" ?

Yes, that's exactly what I meant. You can tell because I didn't present any examples. And because one issue invalidates everything else. And I didn't express my opinion on this issue up thread.

Re: Let's Encrypt is down

#142
post #108
post #77

Earlier quoted context omitted.

I moved from Caddy to Traefik ( https://traefik.io/ ) several months ago. Granted, nginx has years (decades?) on some of these newer webserver/reverse-proxies, but I have been happy with all the built in niceties of traefik so far (single binary, etc), and haven't really experienced any negatives.

You're not bit by this? https://github.com/containous/traefik/issues/791

Nope.

Re: Let's Encrypt is down

#143
post #112

Earlier quoted context omitted.

RFC 6066 specifies that you can only have one certificate in an OCSP response - as with intermediate certs you need to be able to respond with a chain, this does not work. RFC 6961 defines a multiple response capability, but my understanding is that currently this is not sufficiently widely implemented to be useful yet.

Thanks! I thought it's enough if the stapled response contains information only about the intermediate cert, and the browser would accept that as good enough, if the chain it got in the handshake is valid. https://bugzilla.mozilla.org/show_bug.cgi?id=611836 - this looks pretty abandoned (last comment 3 years ago) :/ and I found no bug for Chrome.

Chrome, as far as I know, does not do OCSP - https://www.imperialviolet.org/2012/02/05/crlsets.html

Re: Let's Encrypt is down

#144

Earlier quoted context omitted.

Then wait until you see how caddy handles fully qualified domain names! Caddy will just refuse to even handle them. Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended. Try out: https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ redirects to the relative d…

Is there a bug for this? That's out of spec

I submitted the issue for it (https://github.com/mholt/caddy/issues/1632) and it was immediately closed. Reason (after asking): we only keep issues open that are on our TODO list.

Re: Let's Encrypt is down

#145
post #55

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our…

Thank you for this preliminary report. I just want to say you are doing great work and a tremendous service to the public. We tolerate a few hiccups. And as always when something goes wrong, it is always more than one problem.

Re: Let's Encrypt is down

#146
post #140

Earlier quoted context omitted.

They probably are . But some applications will decode the percent-encoding too early in the process of normalizing, security-escaping, and processing the URL. Encoded slashes in URLs are problematic [1][2][3][4][5]. [1] https://httpd.apache.org/docs/2.4/mod/core.html#allowencoded... [2] http://stackoverflow.com/questions/3235219/urlencoded-forwar... [3] http://codeinthehole.com/tips/django-nginx-wsgi-and-encoded-...…

I guess they most be using something in front of the code [0] (which do document the double slash issue). Probably should have used PathUnescape[1] on line 185 though. [0] https://github.com/letsencrypt/boulder/blob/master/vendor/gi... [1] https://golang.org/pkg/net/url/#PathUnescape PS. It does seem like a pretty bad idea to put data in the path instead of the query.

This is the pull request that triggered today's issue [1]. They introduced a custom object that overrides Go's default ServeMux, such that their code won't collapse adjacent slashes, unlike Go's default.

They did this because previously, they used Go's default, which collapsed adjacent slashes -- this broke stuff until they troubleshooted it [2] and discovered that Cloudflare's OCSP responder, which they were using, actually documents that you shouldn't use the default ServeMux [3]. The commit that led to that condition only went in a month prior [4]. This is similar to an issue they had two years ago [9] that seems to have started this all.

Now the proposal is to strip the "leading slash" from the {ocsp-request} component of the incoming URI "{ocsp-uri}/{ocsp-request}" [5], but far better would be to perform path canonicalization on the {ocsp-uri} but not the {ocsp-request}. But it looks like they're relying on http.StripPrefix [6], which is an idiomatic Go way of hosting a server out of a subpath and returning 404 on any request not matching the prefix; this will be problematic without additional processing that gets slashes out of places they shouldn't be, while leaving alone where they should.

For more fun with slashes and OCSP, see this source code for NSS [7], this bugzilla issue [8], this code [10], and this mailing list thread [11].

[1] https://github.com/letsencrypt/boulder/pull/2748 [2] https://github.com/letsencrypt/boulder/issues/2728 [3] https://github.com/cloudflare/cfssl/blob/master/ocsp/respond... [4] https://github.com/letsencrypt/boulder/pull/2689/files [5] https://github.com/letsencrypt/boulder/issues/2774 [6] https://golang.org/src/net/http/server.go and search for "func StripPrefix" [7] https://chromium.googlesource.com/chromium/third_party/nss/+... and search for "slash" [8] https://bugzilla.mozilla.org/show_bug.cgi?id=1010594 [9] https://github.com/letsencrypt/boulder/issues/884 [10] https://github.com/r509/r509-ocsp-responder/blob/master/lib/... [11] https://sourceforge.net/p/openca/mailman/message/31630541/

Re: Let's Encrypt is down

#149
post #58
post #46

Earlier quoted context omitted.

Not sure how this works. I have OCSP stapling turned on in Apache and Firefox wouldn't load my page when Let's Encrypt OCSP servers went down. My monitoring shows that last stapled response had 4 days of validity left. So it seems that Apache immediately threw away cached OCSP responses.

For what it's worth, Caddy is the only server that will locally cache the staples (and manage them) automatically. In other words, Caddy sites were not affected by this OCSP downtime.

While we're on the topic: https://github.com/mholt/caddy/issues/1680

Re: Let's Encrypt is down

#150
post #58
post #46

Earlier quoted context omitted.

Not sure how this works. I have OCSP stapling turned on in Apache and Firefox wouldn't load my page when Let's Encrypt OCSP servers went down. My monitoring shows that last stapled response had 4 days of validity left. So it seems that Apache immediately threw away cached OCSP responses.

For what it's worth, Caddy is the only server that will locally cache the staples (and manage them) automatically. In other words, Caddy sites were not affected by this OCSP downtime.

While we're on the topic: https://github.com/mholt/caddy/issues/1680
Post reply on HN