Live data from Hacker News

Let's Encrypt is down

letsencrypt.status.io

41–50 of 167 posts

Re: Let's Encrypt is down

#41

I think LE is a huge boon to the internet. But I would really love for someone like Amazon, Google, Facebook, or Microsoft to set up a separate provider that implements the same thing. Redundancy is super important here and clearly just one organization can't guarantee 100% uptime.

I'd vote for Gandi.net or Github to do it. FB definitely not, MS rather not, Google not if I could avoid it.

Amazon maybe.

Re: Let's Encrypt is down

#42

Nothing against letsencrypt but dependencies on services to be online is fragile and will break. Their 90 day limit makes it worse. Saying its for security is like saying 1 or 3 year certs are somehow insecure which is not the case. It's one more headache for an admin to think about even if automated. We really should reexamine the CA system. Self signed certs should have more value than they currently do, and identi…

Long lived certs aren't necessarily insecure but revocation is a real issue that doesn't have a good solution outside of short cert lifetimes. People don't actually revoke their certs because it takes effort and costs money, browsers don't reliably check revocation lists if they do at all, and checking for revocations is bad for performance and privacy.

Re: Let's Encrypt is down

#43

Earlier quoted context omitted.

If you use one of the myriad of automated tools for using LE, you will get your cert renewed as early as 30 days before it expires. So right now the issue should only be with new domains getting certs. If you renew LE certs manually, first what is wrong with you and don't you like yourself? Second, at that point it's no different than NameCheap going down and you getting your cert from 1and1 instead.

Wish someone would write one of those automated tools for Google App Engine-hosted apps. That renewal process is exactly like regex. Once every three months I need it, and have to spend an hour re-learning it from scratch.

Apparently they're working on that: https://issuetracker.google.com/issues/35900034 (For some reason it seems you need to sign in to view this bug; not sure why.)

Re: Let's Encrypt is down

#44

I think LE is a huge boon to the internet. But I would really love for someone like Amazon, Google, Facebook, or Microsoft to set up a separate provider that implements the same thing. Redundancy is super important here and clearly just one organization can't guarantee 100% uptime.

Amazon does, though limited to its own services (which is, frankly, to be expected). AWS Certificate Manager

ACM is kind of limiting though. You can only use it with 4 of AWS's services none of which is ec2 directly which limits use cases I would think.

Re: Let's Encrypt is down

#45
post #33

Nothing against letsencrypt but dependencies on services to be online is fragile and will break. Their 90 day limit makes it worse. Saying its for security is like saying 1 or 3 year certs are somehow insecure which is not the case. It's one more headache for an admin to think about even if automated. We really should reexamine the CA system. Self signed certs should have more value than they currently do, and identi…

OK done: we move to self signed certs. - Someone connects to wifi. - The wifi gives a DNS server. - The DNS server says some IP is foo.com. - foo.com isn't actually the foo.com you expect [1], but it's got a self signed DV cert so you connect to it, and give some bad person your data. That's why we don't more to self signed certs. [1] Of course, if you want to assert foo.com is actually the 'Foo, Inc' you were expect…

TLSA DNS records coupled with DNSSEC could eliminate the need for CAs but we still have a long way to go until we can rely on it.

Re: Let's Encrypt is down

#46
post #37
post #28

Earlier quoted context omitted.

What's the typical validity period for OCSP responses with Let's Encrypt? Shouldn't the stapled responses continue working for at least a couple hours even after Let's Encrypt goes down?

1 week, so most servers likely won't be affected unless the outage goes on for a really long time.

Not sure how this works.

I have OCSP stapling turned on in Apache and Firefox wouldn't load my page when Let's Encrypt OCSP servers went down.

My monitoring shows that last stapled response had 4 days of validity left. So it seems that Apache immediately threw away cached OCSP responses.

Re: Let's Encrypt is down

#47
post #46
post #37

Earlier quoted context omitted.

1 week, so most servers likely won't be affected unless the outage goes on for a really long time.

Not sure how this works. I have OCSP stapling turned on in Apache and Firefox wouldn't load my page when Let's Encrypt OCSP servers went down. My monitoring shows that last stapled response had 4 days of validity left. So it seems that Apache immediately threw away cached OCSP responses.

Yeah, seems like Apache handles OCSP server outages pretty poorly. See: https://news.ycombinator.com/item?id=14375334

Re: Let's Encrypt is down

#48
post #19

Earlier quoted context omitted.

That's going to be a limiter for adoption. Hopefully @mholt reconsiders. Update: Mholt pushed a change where caddy only refuses to start if the cert is expiring in 7 days or less. https://github.com/mholt/caddy/commit/410ece831f26c61d392e0e...

Hm, yeah I hope so too :/ Been using Caddy in prod for a year now, this issue, rare as it may be, could single-handedly get me back on nginx. Having the server be unable to start through circumstances outside of the system's control is just such a huge no.

Why did you switch away from nginx?

Re: Let's Encrypt is down

#49
post #29

So in case this helps anyone, I had people complaining about strange OCSP errors all over the morning coming from my server (using apache httpd). It turns out apache does practically everything to behave as dumb as possible in case of OCSP downtimes. If the OCSP sends an error it will send the error as a stapled OCSP reply (instead of using an old, still valid OCSP reply). You can't make it behave sane here, but you…

additionally, error responses from the OCSP servers are cached 600 seconds - use SSLStaplingErrorCacheTimeout to lower this to a more sane value: https://httpd.apache.org/docs/trunk/mod/mod_ssl.html#sslstap...

Re: Let's Encrypt is down

#50

Was fun finding this out during a random server cycle. Turns out, Caddy doesn't appreciate the ACME server being down, and refuses to start :) https://github.com/mholt/caddy/issues/1680

Wow @ that close comment: > So, this is not a bug and all is working as intended. Caddy folks had better never restart the caddy service (or server) while LE happens to be down, even if you already have a valid cert!

Caddy restarts gracefully with zero downtime. If you're killing the process and starting one anew, you're doing it wrong. Use signal USR1 to gracefully apply new configuration changes. Failed reloads fall back to the current working configuration without any changes or downtime. https://caddyserver.com/docs/cli#usr1
Post reply on HN