Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Encrypt is down

letsencrypt.status.io

Post reply on HN
Print view View original

101–110 of 167 posts

Re: Let's Encrypt is down

#101

Post by stephenr »

scrollaway wrote: post #91
stephenr wrote: post #86

Earlier quoted context omitted.

> by running Caddy instead, I have one less piece to monitor and worry about No, with Caddy you have several vaguely related pieces glued together with superglue. > The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server. I think this thread is pretty much proof that that approach will bite you in the ass.

> I think this thread is pretty much proof that that approach will bite you in the ass. What bit me here is the fact that I'm running alpha software instead of a battle-tested web server; I'm doing so willingly, with full awareness of the risks that that entails. Drawing the conclusion you did from the variables at play is shortsighted. If anything bites people in the ass, it's prejudice and shortsightedness. I would…

> What bit me here is the fact that I'm running alpha software

This wasn't caused by a bug. This was a deliberate decision to fail to start if the certificates on-disk were days away from expiring and the CA can't be contacted.

> Drawing the conclusion you did from the variables at play is shortsighted

Using caddy is the web-server-stack equivalent of "putting all your eggs in one basket". If one thing about it isn't working the way you want, you have to either a) replace it completely or b) work out how to disable the bit that's not working how you want, and replace that part of it.

> Drawing the conclusion you did from the variables at play is shortsighted

- People use a piece of software that serves as both ACME TLS certificate client and web server

- Said software by design won't start if the CA can't be contacted 30-days out from expiry

The conclusion I drew is that such integration leave the operator with less control than if they followed a separation-of-concerns approach, and left web serving to a web server, and TLS certificate renewal to an ACME client. The former doesn't need to care about how old the certificates are, just use what it's given.

Re: Let's Encrypt is down

#102

Post by JoshTriplett »

lilei2 wrote: post #93
eric_b wrote: post #88

Earlier quoted context omitted.

As I was reading the first few sentences, describing the slash collapsing, I was thinking to myself "oh no, I hope they don't just 'fix the glitch'". That behavior is so old and pervasive on the web, about the last thing I would have tried is turning off slash collapsing.

I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? (I guess this is more a question for the parent)

> I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place?

base64 uses 64 characters: A-Za-z0-9 (62) and two symbols, commonly '/' and '+'. (As well as a third symbol, '=', used at the end to handle padding.) That would work in a URI, most of the time, except if you happened to have a base64 encoding with two '/' next to each other.

A common fix for using base64 in URIs involves substituting a different pair of symbols instead of '/' and '+'.

Re: Let's Encrypt is down

#103

Post by stephenr »

scrollaway wrote: post #94
stephenr wrote: post #90

Earlier quoted context omitted.

> That's a bit of a straw man From https://caddyserver.com : > The Most Beloved Server They started the hyperbolic claims, not me.

I have no idea how this is relevant to the conversation, nor who said anything about hyperbolic claims. You're claiming "caddy does everything". As opposed to what? If you're running apache or nginx, your server does far more than caddy, so you're quite simply mistaken.

> You're claiming "caddy does everything". As opposed to what?

Serving content over http(s), and obtaining TLS certificates are two very different tasks.

> If you're running apache or nginx, your server does far more than caddy

Far more, that is directly related to serving content over http/https.

Re: Let's Encrypt is down

#104

Post by tlrobinson »

jaas wrote: post #55

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our…

1. Spec question: Why does the request need to be both base 64 and URL encoded? Why not just URL encoded? Only reason I can think of is for shorter/prettier URLs?

Or why not just use base 64 with the URL safe alphabet: https://tools.ietf.org/html/rfc4648#page-7

2. Implementation question: Shouldn't the slashes be URL encoded as "%2F"? "url/ABC/DEF" could mean "url" + "ABC/DEF" or "url/ABC" + "DEF". Multiple slashes are collapsed by default because path components shouldn't contain them.

Re: Let's Encrypt is down

#105

Post by scrollaway »

stephenr wrote: post #101
scrollaway wrote: post #91

Earlier quoted context omitted.

> I think this thread is pretty much proof that that approach will bite you in the ass. What bit me here is the fact that I'm running alpha software instead of a battle-tested web server; I'm doing so willingly, with full awareness of the risks that that entails. Drawing the conclusion you did from the variables at play is shortsighted. If anything bites people in the ass, it's prejudice and shortsightedness. I would…

> What bit me here is the fact that I'm running alpha software This wasn't caused by a bug. This was a deliberate decision to fail to start if the certificates on-disk were days away from expiring and the CA can't be contacted. > Drawing the conclusion you did from the variables at play is shortsighted Using caddy is the web-server-stack equivalent of "putting all your eggs in one basket". If one thing about it isn't…

You're drawing conclusions from unintended behaviour, which has now changed (and a release has been issued).

Re: Let's Encrypt is down

#106

Post by niftich »

r1ch wrote: post #78
TechTechTech wrote: post #70

Earlier quoted context omitted.

Just a quick question. Does this mean that if your OCSP servers were to go down, a lot of SSL enabled websites and applications will stop working? Seems like a serious single point of failure for modern day internet. I was always under the assumption that clients do not have to contact the CA (every time?) before a TLS handshake takes place. OCSP Stapling seems to be the way to mitigate this problem, but not all web…

I'm pretty sure revocation checks like CRL and OCSP all fail-open (they still allow the connection if contacting the revocation server fails).

Some have argued that this is why CRL and (especially) OCSP are useless pieces of security theater: they don't actually protect against a crafted attack because they fail-open in the very situations that a determined adversary can trigger, so they only "protect" in situations where no real threat exists. It's simply feel-good bookkeeping.

Adam Langley, working on Google Chrome [1][2][3], has been very vocal about OCSP's faults, and Chrome began using its own auto-update to ship an aggregate of revocations of high-value certs directly to browsers out-of-band. Despite this being another famous instance of Chrome going against the grain of other browser vendors, I believe this was the correct solution: offering better protection for a curated subset of sites vs. pretending to -- but not actually -- protecting all sites.

[1] https://www.imperialviolet.org/2012/02/05/crlsets.html [2] https://www.imperialviolet.org/2014/04/19/revchecking.html [3] http://www.zdnet.com/article/chrome-does-certificate-revocat...

Re: Let's Encrypt is down

#107

Post by tlrobinson »

JoshTriplett wrote: post #102
lilei2 wrote: post #93

Earlier quoted context omitted.

I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? (I guess this is more a question for the parent)

> I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? base64 uses 64 characters: A-Za-z0-9 (62) and two symbols, commonly '/' and '+'. (As well as a third symbol, '=', used at the end to handle padding.) That would work in a URI, most of the time, except if you happ…

The spec says the base 64 should be URL encoded, so why aren't the slashes turned into "%2F"?

Re: Let's Encrypt is down

#108

Post by mholt »

e_d_e_v wrote: post #77
GiorgioG wrote: post #51

Earlier quoted context omitted.

I'm done with Caddy - this is security theater at its finest. Back to nginx we go. Why did I pick Caddy, because it was simple/easy/fast-to-setup.

I moved from Caddy to Traefik ( https://traefik.io/ ) several months ago. Granted, nginx has years (decades?) on some of these newer webserver/reverse-proxies, but I have been happy with all the built in niceties of traefik so far (single binary, etc), and haven't really experienced any negatives.

You're not bit by this? https://github.com/containous/traefik/issues/791

Re: Let's Encrypt is down

#109

Post by kuschku »

discreditable wrote: post #13
scrollaway wrote: post #2

Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.com/mholt/caddy/issues/1680

Wow @ that close comment: > So, this is not a bug and all is working as intended. Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!

Then wait until you see how caddy handles fully qualified domain names!

Caddy will just refuse to even handle them.

Every single other server on this planet handles them properly, but caddy doesn’t – and mholt considers that working as intended.

Try out: https://www.google.co.uk./ https://www.microsoft.com./en-us/ https://www.amazon.com./ serve the page directly; https://www.facebook.com./ redirects to the relative domain

and then https://caddyserver.com./ (That said, traefik is equally dumb, as seen with https://traefik.io./ )

Re: Let's Encrypt is down

#110

Post by IgorPartola »

Bartweiss wrote: post #39
IgorPartola wrote: post #8

Earlier quoted context omitted.

If you use one of the myriad of automated tools for using LE, you will get your cert renewed as early as 30 days before it expires. So right now the issue should only be with new domains getting certs. If you renew LE certs manually, first what is wrong with you and don't you like yourself? Second, at that point it's no different than NameCheap going down and you getting your cert from 1and1 instead.

> So right now the issue should only be with new domains getting certs. Certainly should have been, but it looks like a lot of libraries are choking on the outage even if the old cert is 100% valid.

I use https://github.com/lukas2511/dehydrated which is a bash script for doing this. It doesn't choke because it's just a cron job.

I also use dokku's LE plugin which is... also a shell script corn job. Maybe that's the way to do this. I know Caddy is an exception case for this.

Post reply on HN