Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Encrypt is down

letsencrypt.status.io

Post reply on HN
Print view View original

111–120 of 167 posts

Re: Let's Encrypt is down

#111

Post by discreditable »

danielbln wrote: post #36
scrollaway wrote: post #32

Earlier quoted context omitted.

Hm, yeah I hope so too :/ Been using Caddy in prod for a year now, this issue, rare as it may be, could single-handedly get me back on nginx. Having the server be unable to start through circumstances outside of the system's control is just such a huge no.

Same here, was very pleased with Caddy so far, but it being tightly coupled to LE being up, despite having certs cached is a no-go for our production systems and if this stays like that, would make me go back to nginx for the services I've used caddy so far. edit: looks like the dev added a fix to only refuse the start if the cached certs are dangerously close to expire. that satisfies me and I'll be continue to be u…

Call me crazy but I think it's a little silly for an https server to refuse to start just because its certificate is invalid. I recognize that some folks might like that failure mode however.

Re: Let's Encrypt is down

#112

Post by pas »

jfindley wrote: post #97
pas wrote: post #92

Earlier quoted context omitted.

Could you elaborate on why they don't work with intermediate certs?

RFC 6066 specifies that you can only have one certificate in an OCSP response - as with intermediate certs you need to be able to respond with a chain, this does not work. RFC 6961 defines a multiple response capability, but my understanding is that currently this is not sufficiently widely implemented to be useful yet.

Thanks! I thought it's enough if the stapled response contains information only about the intermediate cert, and the browser would accept that as good enough, if the chain it got in the handshake is valid.

https://bugzilla.mozilla.org/show_bug.cgi?id=611836 - this looks pretty abandoned (last comment 3 years ago) :/

and I found no bug for Chrome.

Re: Let's Encrypt is down

#113

Post by corford »

CaliforniaKarl wrote: post #59

It seems that Let's Encrypt is back up. I think, in the past, mods have put an extra down-weight on "X is down" stories, once 'X' is back up. Since this discussion now has interesting stuff related to Let's Encrypt—and products which use Let's Encrypt—I hope the mods are willing to forgo the down weight, and instead just change the post title to something like "Let's Encrypt Was Down".

I'm still getting 504s and timeouts from https://acme-v01.api.letsencrypt.org/acme/new-reg and https://acme-v01.api.letsencrypt.org/directory (trying from Lisbon, PT)

Edit: Akamai is issuing the 504 when hitting https://acme-v01.api.letsencrypt.org/acme/new-reg so I guess the origin servers are still overloaded...?

Re: Let's Encrypt is down

#114

Post by spiderfarmer »

tlrobinson wrote: post #107
JoshTriplett wrote: post #102

Earlier quoted context omitted.

> I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? base64 uses 64 characters: A-Za-z0-9 (62) and two symbols, commonly '/' and '+'. (As well as a third symbol, '=', used at the end to handle padding.) That would work in a URI, most of the time, except if you happ…

The spec says the base 64 should be URL encoded, so why aren't the slashes turned into "%2F"?

This is what I don't understand either.

Re: Let's Encrypt is down

#115

Post by wolf550e »

JoshTriplett wrote: post #102
lilei2 wrote: post #93

Earlier quoted context omitted.

I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? (I guess this is more a question for the parent)

> I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? base64 uses 64 characters: A-Za-z0-9 (62) and two symbols, commonly '/' and '+'. (As well as a third symbol, '=', used at the end to handle padding.) That would work in a URI, most of the time, except if you happ…

for reference, see "base64url" in this table: https://en.wikipedia.org/wiki/Base64#Variants_summary_table

Re: Let's Encrypt is down

#116

Post by zkms »

niftich wrote: post #106
r1ch wrote: post #78

Earlier quoted context omitted.

I'm pretty sure revocation checks like CRL and OCSP all fail-open (they still allow the connection if contacting the revocation server fails).

Some have argued that this is why CRL and (especially) OCSP are useless pieces of security theater: they don't actually protect against a crafted attack because they fail-open in the very situations that a determined adversary can trigger, so they only "protect" in situations where no real threat exists. It's simply feel-good bookkeeping. Adam Langley, working on Google Chrome [1][2][3], has been very vocal about OCS…

> I believe this was the correct solution: offering better protection for a curated subset of sites vs. pretending to -- but not actually -- protecting all sites.

I concur, but note that it is possible to do better and offer better revocation protection for all sites, with low bandwidth/storage costs: http://www.ccs.neu.edu/home/cbw/static/pdf/larisch-oakland17...

Re: Let's Encrypt is down

#117

Post by yjftsjthsd-h »

kuschku wrote: post #109
discreditable wrote: post #13

Earlier quoted context omitted.

Wow @ that close comment: > So, this is not a bug and all is working as intended. Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!

Then wait until you see how caddy handles fully qualified domain names! Caddy will just refuse to even handle them. Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended. Try out: https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ redirects to the relative d…

Is there a bug for this? That's out of spec

Re: Let's Encrypt is down

#118

Post by yjftsjthsd-h »

jacquesm wrote: post #41
IgorPartola wrote: post #10

I think LE is a huge boon to the internet. But I would really love for someone like Amazon, Google, Facebook, or Microsoft to set up a separate provider that implements the same thing. Redundancy is super important here and clearly just one organization can't guarantee 100% uptime.

I'd vote for Gandi.net or Github to do it. FB definitely not, MS rather not, Google not if I could avoid it. Amazon maybe.

What's the threat model? I don't trust FB either, but with CA transparency and CAA it seems safe enough.

Re: Let's Encrypt is down

#119

Post by niftich »

tlrobinson wrote: post #104
jaas wrote: post #55

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our…

1. Spec question: Why does the request need to be both base 64 and URL encoded? Why not just URL encoded? Only reason I can think of is for shorter/prettier URLs? Or why not just use base 64 with the URL safe alphabet: https://tools.ietf.org/html/rfc4648#page-7 2. Implementation question: Shouldn't the slashes be URL encoded as "%2F"? "url/ABC/DEF" could mean "url" + "ABC/DEF" or "url/ABC" + "DEF". Multiple slashes a…

Josh referred to RFC 6960 Appendix A, but his post didn't make it apparent that his description of is an exact quote from the spec [1]:

An OCSP request using the GET method is constructed as follows:

GET {url}/{url-encoding of base-64 encoding of the DER encoding of the OCSPRequest}

This is shamefully imprecise for an RFC, not even referencing the relevant specs for each.

--

To answer your first question, Base64 is needed because DER is binary, and URIs are defined in terms of characters -- and it's an exercise left to the reader [2] if you can somehow get from one to the other reliably. It's also an exercise for every other reader you're trying to interoperate it, so the common practice is to make an explicit conversion before you get to this stage. Base64 takes care of this by transforming the binary data to US-ASCII, of which UTF-8 is a superset, and URIs operate on UTF-8 characters.

But "vanilla" base64 can produce three characters which are reserved characters in URIs: the slash, the plus, and equals [8]. These need to be percent-encoded because if they are used directly in URIs, they have special meaning.

Of course, if the OCSP RFC had just specified base64url encoding [9], a widely used variant which swaps the slash with an underline, the plus with a minus, and allows for the omission of padding that's denoted by equals signs, the double-encoding wouldn't be needed, because none of those characters are reserved in URIs.

--

To answer your second question, slashes in URIs are fun. Though the most recent URI RFC goes through elaborate rules on when you're supposed to encode and decode [3] and what's supposed to be interpreted how, at the end of the day the URI is somehow consumed as an input to some other process where different rules may apply [5][6].

One of those different, customary "rules" is that percent-encoded slashes are just maliciously trying to path outside of the directory, so most webservers shut this down. Apache is one of the few that allows you to tune what to do in this case [7].

[1] https://tools.ietf.org/html/rfc6960#appendix-A.1 [2] https://tools.ietf.org/html/rfc3986#section-2.5 [3] https://tools.ietf.org/html/rfc3986#section-2.4 [4] https://tools.ietf.org/html/rfc3986#section-3.3 [5] https://tools.ietf.org/html/rfc3986#section-7.2 [6] https://tools.ietf.org/html/rfc3986#section-7.3 [7] http://httpd.apache.org/docs/2.4/mod/core.html#allowencodeds... [8] https://tools.ietf.org/html/rfc4648#section-4 [9] https://tools.ietf.org/html/rfc4648#section-5

Re: Let's Encrypt is down

#120

Post by theprop »

We've been using LetsEncrypt on dozens of our servers for several months now and it's worked flawlessly both in set-up and in operation!! Set up was quite easy thanks to A LOT of work by the team there.

To make HTTPS simple and free to set up is a FANTASTIC mission & the team has overall built a SUPERB system. Congratulations & thanks for addressing this issue quickly, looks like it's already solved, good work!

Post reply on HN