Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Encrypt is down

letsencrypt.status.io

Post reply on HN
Print view View original

131–140 of 167 posts

Re: Let's Encrypt is down

#131

Post by shevy »

jaas wrote: post #55

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our…

yo people - you guys really need to handle slashes properly!

They are quite important for www-stuff you know.

"a majority of the OCSP requests we were receiving were prepended with a slash"

Everything is unsafe - one has to make sure that external data is converted into a sane format internally; and to never assume that external input can be safe!

Re: Let's Encrypt is down

#132

Post by hinkley »

r1ch wrote: post #78
TechTechTech wrote: post #70

Earlier quoted context omitted.

Just a quick question. Does this mean that if your OCSP servers were to go down, a lot of SSL enabled websites and applications will stop working? Seems like a serious single point of failure for modern day internet. I was always under the assumption that clients do not have to contact the CA (every time?) before a TLS handshake takes place. OCSP Stapling seems to be the way to mitigate this problem, but not all web…

I'm pretty sure revocation checks like CRL and OCSP all fail-open (they still allow the connection if contacting the revocation server fails).

The one app chain (complex code signing) I worked on with OSCP, we defaulted to failsafe, but it could be overridden in the 'main' (enterprise CMS) app. The installer required OSCP or wouldn't install.

Basically the first and last mile were hard fails but everything in between was advisory if the signature checked out.

Re: Let's Encrypt is down

#134

Post by eropple »

e_d_e_v wrote: post #98
eropple wrote: post #85

Earlier quoted context omitted.

How so? In Chef: notifies :reload, "service[caddy]", :delayed

The idea here is that the server can't be scaled up or down. I suggest googling "cattle vs pets". If you know how to make a single process scale horizontally across additional hosts or scale up in alternative datacenters with a chef service notification, I'd pay you money to tell me how.

So, um: devops and cloud architecture is my job. I think I know what "cattle vs pets" is referring to. Nothing in this thread has anything to do with one versus many processes or one versus many nodes, nor does it have anything to do with manual configuration of anything. Rewrite the configuration based in orchestration data or, in extremity, upload a new version of a cookbook that handles that new site; the hypothetical sixth site is added and the service kicked over without human interaction. Scaling is an orthogonal concern.

Re: Let's Encrypt is down

#135

Post by gommm »

scrollaway wrote: post #105
stephenr wrote: post #101

Earlier quoted context omitted.

> What bit me here is the fact that I'm running alpha software This wasn't caused by a bug. This was a deliberate decision to fail to start if the certificates on-disk were days away from expiring and the CA can't be contacted. > Drawing the conclusion you did from the variables at play is shortsighted Using caddy is the web-server-stack equivalent of "putting all your eggs in one basket". If one thing about it isn't…

You're drawing conclusions from unintended behaviour, which has now changed (and a release has been issued).

I still think that refusing to start if the cert expires in 7 days or less is still an issue if Let's Encrypt is down.

There should be at most a warning but it should start. Otherwise you end up with an external dependency that can cause your web server to not start through no fault of your own.

Re: Let's Encrypt is down

#136

Post by the_mitsuhiko »

tlrobinson wrote: post #107
JoshTriplett wrote: post #102

Earlier quoted context omitted.

> I'm not sure I understand how slash collapsing is affecting this. Slash is a reserved character and presumably if it the data was correctly encoded should never have ended up in the URI in the first place? base64 uses 64 characters: A-Za-z0-9 (62) and two symbols, commonly '/' and '+'. (As well as a third symbol, '=', used at the end to handle padding.) That would work in a URI, most of the time, except if you happ…

The spec says the base 64 should be URL encoded, so why aren't the slashes turned into "%2F"?

It does not matter. Due to how servers and most apps need to handle URLs they are decoded very early in the process (eg: %2F and %2f need to be the same for instance).

Re: Let's Encrypt is down

#137

Post by stephenr »

scrollaway wrote: post #105
stephenr wrote: post #101

Earlier quoted context omitted.

> What bit me here is the fact that I'm running alpha software This wasn't caused by a bug. This was a deliberate decision to fail to start if the certificates on-disk were days away from expiring and the CA can't be contacted. > Drawing the conclusion you did from the variables at play is shortsighted Using caddy is the web-server-stack equivalent of "putting all your eggs in one basket". If one thing about it isn't…

You're drawing conclusions from unintended behaviour, which has now changed (and a release has been issued).

> unintended behaviour

Ahem. https://github.com/mholt/caddy/issues/1680#issuecomment-3026...

Emphasis mine:

> So, this is not a bug and all is working as intended.

Re: Let's Encrypt is down

#138

Post by stephenr »

tyingq wrote: post #79
fb03 wrote: post #66

Earlier quoted context omitted.

I am running nginx reverse-proxying to a python API right now. Dealing with certificate renewal is a matter of running a daily cronjob issuing 'certbot renew'. If it works it replaces the fullchain.pem certificate, and that's it, easy peasy. Am I missing something?

The simple config syntax and sane defaults is nice in Caddy. For example, 3 lines of config nets a https server with http/2 support and an A rating with Qualsys for ssl setup.

> sane defaults

Failure to start if a CA is down is a "sane default" ?

Re: Let's Encrypt is down

#139

Post by hdhzy »

jaas wrote: post #55

Josh from Let's Encrypt here. First, my apologies for the trouble this has caused. I want to offer people here an early root cause analysis. I say early because we have not entirely completed our investigation or a post-mortem. OCSP requests that use the GET method use standard base64 encoding, which can contain two slashes one after another. While debugging why a small number of OCSP requests consistently failed our…

That means OCSP Must-Staple extension [0] can be very dangerous given bad implementation (like Apache).

[0]: https://scotthelme.co.uk/ocsp-must-staple/

Re: Let's Encrypt is down

#140

Post by lilei2 »

niftich wrote: post #130
tlrobinson wrote: post #107

Earlier quoted context omitted.

The spec says the base 64 should be URL encoded, so why aren't the slashes turned into "%2F"?

They probably are . But some applications will decode the percent-encoding too early in the process of normalizing, security-escaping, and processing the URL. Encoded slashes in URLs are problematic [1][2][3][4][5]. [1] https://httpd.apache.org/docs/2.4/mod/core.html#allowencoded... [2] http://stackoverflow.com/questions/3235219/urlencoded-forwar... [3] http://codeinthehole.com/tips/django-nginx-wsgi-and-encoded-...…

I guess they most be using something in front of the code [0] (which do document the double slash issue). Probably should have used PathUnescape[1] on line 185 though.

[0] https://github.com/letsencrypt/boulder/blob/master/vendor/gi... [1] https://golang.org/pkg/net/url/#PathUnescape

PS. It does seem like a pretty bad idea to put data in the path instead of the query.

Post reply on HN