Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Encrypt is down

letsencrypt.status.io

Post reply on HN
Print view View original

81–90 of 167 posts

Re: Let's Encrypt is down

#81

Post by sleepychu »

majewsky wrote: post #71
atonse wrote: post #65

Earlier quoted context omitted.

This misses the overall point. If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. If the 6th site doesn't work (for example, if DNS didn't resolve for lets encrypt), the other 5 sites which were working before the restart, all fail to start as caddy completely crashes. This is basic resiliency you'd expect from your web server. Why should the other 5 sites fail to start if their configs are co…

> If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)

OK, there's a 0-day patch for OpenSSL, without it my users are vulnerable to RCE. Why can't I restart if LE's ACME is down?

Re: Let's Encrypt is down

#82

Post by jfindley »

TechTechTech wrote: post #70
jaas wrote: post #55

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our…

Just a quick question. Does this mean that if your OCSP servers were to go down, a lot of SSL enabled websites and applications will stop working? Seems like a serious single point of failure for modern day internet. I was always under the assumption that clients do not have to contact the CA (every time?) before a TLS handshake takes place. OCSP Stapling seems to be the way to mitigate this problem, but not all web…

This is exactly the problem with OCSP. There's no way to tell if the remote server is down, or if a malicious actor sitting in your path is blocking it. So your browser can either a) make it super easy for all your OCSP-using sites to appear down, which will encourage users to use other, non-OCSP, sites, or b) silently fail, which makes the entire exercise pointless.

Stapling only partially mitigates this, as it doesn't currently work with intermediate certs, and at this point most sites have at least one intermediate cert.

Re: Let's Encrypt is down

#83

Post by stephenr »

scrollaway wrote: post #2

Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.com/mholt/caddy/issues/1680

This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this.

Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered.

What exactly do you do when your look-ma-no-hands server won't even start?

Edit: maybe "all-things-to-all-people" was the wrong term to use here.

Re: Let's Encrypt is down

#84

Post by scrollaway »

JshWright wrote: post #75
scrollaway wrote: post #74

Earlier quoted context omitted.

And by running Caddy instead, I have one less piece to monitor and worry about. The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.

> The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server. That's an odd thing to say in this particular conversation thread... Why would you want to tightly couple your webserver to the availability of another service provider?

It's not tightly coupled any more than it would be with certbot. The issue I filed is an issue because it's unexpected behaviour (despite early closure, it has actually now been fixed).

Re: Let's Encrypt is down

#85

Post by eropple »

e_d_e_v wrote: post #76
majewsky wrote: post #71

Earlier quoted context omitted.

> If I am hosting 5 sites on caddy, and add a 6th one, I restart the server. No, you don't. You reload the server, not restart it. Restarting a web server should only be required if you get an upgrade for it (or for OpenSSL etc.)

This goes wholly against most cattle-not-pets devops philosophy though, right?

How so? In Chef:

   notifies :reload, "service[caddy]", :delayed

Re: Let's Encrypt is down

#86

Post by stephenr »

scrollaway wrote: post #74
fb03 wrote: post #66

Earlier quoted context omitted.

I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy. Am I missing something?

And by running Caddy instead, I have one less piece to monitor and worry about. The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.

> by running Caddy instead, I have one less piece to monitor and worry about

No, with Caddy you have several vaguely related pieces glued together with superglue.

> The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.

I think this thread is pretty much proof that that approach will bite you in the ass.

Re: Let's Encrypt is down

#87

Post by scrollaway »

stephenr wrote: post #83
scrollaway wrote: post #2

Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.com/mholt/caddy/issues/1680

This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this. Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered. What exactly do you do when your look-ma-no-hands server won't even start? Edit: maybe "all-things-to-all-people" was the wrong term to use here.

That's a bit of a strawman. Caddy is far more lightweight than apache and even nginx. It just happens to do something they don't do.

Re: Let's Encrypt is down

#88

Post by eric_b »

jaas wrote: post #55

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our…

As I was reading the first few sentences, describing the slash collapsing, I was thinking to myself "oh no, I hope they don't just 'fix the glitch'". That behavior is so old and pervasive on the web, about the last thing I would have tried is turning off slash collapsing.

Re: Let's Encrypt is down

#89

Post by Ajedi32 »

scrollaway wrote: post #2

Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.com/mholt/caddy/issues/1680

Updating to v0.10.3 should fix this problem and allow Caddy to start, provided your cert isn't less than 7 days from expiring: https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...

Re: Let's Encrypt is down

#90

Post by stephenr »

scrollaway wrote: post #87
stephenr wrote: post #83

Earlier quoted context omitted.

This type of thing is why I (and I'm sure others) have literally zero intention of using tools like this. Separation of concerns means you are in control, and using separate layers means you can swap one out when a vulnerability/show-stopper bug is discovered. What exactly do you do when your look-ma-no-hands server won't even start? Edit: maybe "all-things-to-all-people" was the wrong term to use here.

That's a bit of a strawman. Caddy is far more lightweight than apache and even nginx. It just happens to do something they don't do.

> That's a bit of a straw man

From https://caddyserver.com:

> The Most Beloved Server

They started the hyperbolic claims, not me.

Post reply on HN