Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Encrypt is down

letsencrypt.status.io

Post reply on HN
Print view View original

91–100 of 167 posts

Re: Let's Encrypt is down

#91

Post by scrollaway »

stephenr wrote: post #86
scrollaway wrote: post #74

Earlier quoted context omitted.

And by running Caddy instead, I have one less piece to monitor and worry about. The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server.

> by running Caddy instead, I have one less piece to monitor and worry about No, with Caddy you have several vaguely related pieces glued together with superglue. > The way Let's Encrypt works, it makes a lot of sense to have the functionality be part of the web server. I think this thread is pretty much proof that that approach will bite you in the ass.

> I think this thread is pretty much proof that that approach will bite you in the ass.

What bit me here is the fact that I'm running alpha software instead of a battle-tested web server; I'm doing so willingly, with full awareness of the risks that that entails.

Drawing the conclusion you did from the variables at play is shortsighted. If anything bites people in the ass, it's prejudice and shortsightedness. I wouldn't want you handling my ops/infrastructure.

Re: Let's Encrypt is down

#92

Post by pas »

jfindley wrote: post #82
TechTechTech wrote: post #70

Earlier quoted context omitted.

Just a quick question. Does this mean that if your OCSP servers were to go down, a lot of SSL enabled websites and applications will stop working? Seems like a serious single point of failure for modern day internet. I was always under the assumption that clients do not have to contact the CA (every time?) before a TLS handshake takes place. OCSP Stapling seems to be the way to mitigate this problem, but not all web…

This is exactly the problem with OCSP. There's no way to tell if the remote server is down, or if a malicious actor sitting in your path is blocking it. So your browser can either a) make it super easy for all your OCSP-using sites to appear down, which will encourage users to use other, non-OCSP, sites, or b) silently fail, which makes the entire exercise pointless. Stapling only partially mitigates this, as it does…

Could you elaborate on why they don't work with intermediate certs?

Re: Let's Encrypt is down

#93

Post by lilei2 »

eric_b wrote: post #88
jaas wrote: post #55

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our…

As I was reading the first few sentences, describing the slash collapsing, I was thinking to myself "oh no, I hope they don't just 'fix the glitch'". That behavior is so old and pervasive on the web, about the last thing I would have tried is turning off slash collapsing.

I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place?

(I guess this is more a question for the parent)

Re: Let's Encrypt is down

#94

Post by scrollaway »

stephenr wrote: post #90
scrollaway wrote: post #87

Earlier quoted context omitted.

That's a bit of a strawman. Caddy is far more lightweight than apache and even nginx. It just happens to do something they don't do.

> That's a bit of a straw man From https://caddyserver.com : > The Most Beloved Server They started the hyperbolic claims, not me.

I have no idea how this is relevant to the conversation, nor who said anything about hyperbolic claims.

You're claiming "caddy does everything". As opposed to what? If you're running apache or nginx, your server does far more than caddy, so you're quite simply mistaken.

Re: Let's Encrypt is down

#95

Post by alberts00 »

atonse wrote: post #22
tyingq wrote: post #20

Earlier quoted context omitted.

ACM is nice, but it does require the manual step of clicking a link in a verification email.

Yes but it also issues certs for a year, which helps alleviate that email link issue.

If you are running the AWS ACM certificate at the time when the old one is close to expiration it will automatically renew it without user intervention. http://docs.aws.amazon.com/acm/latest/userguide/configure-do...

Re: Let's Encrypt is down

#96

Post by misnome »

lilei2 wrote: post #93
eric_b wrote: post #88

Earlier quoted context omitted.

As I was reading the first few sentences, describing the slash collapsing, I was thinking to myself "oh no, I hope they don't just 'fix the glitch'". That behavior is so old and pervasive on the web, about the last thing I would have tried is turning off slash collapsing.

I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? (I guess this is more a question for the parent)

So, blame the clients and users unfortunate enough to be using an implementation that only works 99.999% of the time?

Re: Let's Encrypt is down

#97

Post by jfindley »

pas wrote: post #92
jfindley wrote: post #82

Earlier quoted context omitted.

This is exactly the problem with OCSP. There's no way to tell if the remote server is down, or if a malicious actor sitting in your path is blocking it. So your browser can either a) make it super easy for all your OCSP-using sites to appear down, which will encourage users to use other, non-OCSP, sites, or b) silently fail, which makes the entire exercise pointless. Stapling only partially mitigates this, as it does…

Could you elaborate on why they don't work with intermediate certs?

RFC 6066 specifies that you can only have one certificate in an OCSP response - as with intermediate certs you need to be able to respond with a chain, this does not work. RFC 6961 defines a multiple response capability, but my understanding is that currently this is not sufficiently widely implemented to be useful yet.

Re: Let's Encrypt is down

#98

Post by e_d_e_v »

eropple wrote: post #85
e_d_e_v wrote: post #76

Earlier quoted context omitted.

This goes wholly against most cattle-not-pets devops philosophy though, right?

How so? In Chef: notifies :reload, "service[caddy]", :delayed

The idea here is that the server can't be scaled up or down. I suggest googling "cattle vs pets". If you know how to make a single process scale horizontally across additional hosts or scale up in alternative datacenters with a chef service notification, I'd pay you money to tell me how.

Re: Let's Encrypt is down

#99

Post by hannob »

avian wrote: post #52
hannob wrote: post #29

So in case this helps anyone, I had people complaining about strange OCSP errors all over the morning coming from my server (using apache httpd). It turns out apache does practically everything to behave as dumb as possible in case of OCSP downtimes. If the OCSP sends an error it will send the error as a stapled OCSP reply (instead of using an old, still valid OCSP reply). You can't make it behave sane here, but you…

Thanks for pointing this out! Going through the documentation, another thing that surprised me was the SSLStaplingStandardCacheTimeout setting. If I understand this correctly, by default Apache will only cache OCSP responses for 1 hour, even if they are still valid for days. I guess increasing this to 1 day or something would make sense as well.

> If I understand this correctly, by default Apache will only cache OCSP responses for 1 hour, even if they are still valid for days. I guess increasing this to 1 day or something would make sense as well.

Yeah, that's another major problem. But increasing that doesn't really fix anything in a reasonable way. There shouldn't be any cache timeout, this option doesn't make any sense. It should cache as long as the reply is valid and replace it in time before it expires.

Re: Let's Encrypt is down

#100

Post by lilei2 »

misnome wrote: post #96
lilei2 wrote: post #93

Earlier quoted context omitted.

I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? (I guess this is more a question for the parent)

So, blame the clients and users unfortunate enough to be using an implementation that only works 99.999% of the time?

Blame? It's more about curiosity.

The poster said "OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another", so I'm wondering if that's that case or it's actually "clients don't encode their base64". I didn't read trough the RFC, but judging by the later statement of "OCSP request using the GET method is constructed as follows 'GET {url}/{url-encoding of base-64 encoding of the DER encoding of the OCSPRequest}'" that seems to be the case.

Post reply on HN