How much of the internet is going to fail because of this?
None, unless someone is renewing their certificates only 2 hours before they expire, which is a dumb thing to do.
Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
61–70 of 97 posts
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#62That's really not good. Fortunately I'm not using any short-lived certificates like the recently announced 6 day certs, so have some breathing room. Without further details, I'd imagine anyone with a short-lived cert is getting a bit sweaty right now. Let's Encrypt has become one of those pieces of critical Internet infrastructure that just quietly hums away in the background, the fact that they've stopped ALL issuan…
Considering the open source nature of Letsencrypt, I wonder what the barriers/costs would be (theoretically) to a wealthy benefactor who wanted to duplicate its server side infrastructure and a core staffing level of persons, and fund a "parallel" equally trusted, alternative entity with a solid governing board. Same general idea how Acton funded the Signal foundation. Somewhere that none of the physical infrastructu…
Google has their own free ACME endpoint: https://pki.goog/
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#63Related Cloudflare issue:
https://www.cloudflarestatus.com/incidents/z3vgxxfvt3yb
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#64Discord is out too right now, probably unrelated though.
Just speculating, but I don't think it's unrelated. Discord heavily utilizes Cloudflare, and Cloudflare uses Let's Encrypt for a certificate issuance. If they happened to have a certificate signing dependency in some operational rollout today, I think it could explain it. Certainly the timing is very correlated.
On my account they always serve Google issued certificates. There is also Let’s encrypt certificate but it is not used though. I guess that’s a fail-safe.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#65This is a compliance incident, we should be issuing again shortly. Update: Issuance is back up. Update: Preliminary incident report: https://bugzilla.mozilla.org/show_bug.cgi?id=2038351
can you update the status page with this information?
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#66Earlier quoted context omitted.
Indeed. "Compliance" can mean some internal audit/monitoring system has tripped and requires in depth investigation and preservation of logging, or it can mean "federal law enforcement with badges are right now standing in our datacenter and/or NOC serving a court order".
At times like this it's worth remembering that message boards strongly favor whatever narrative is going to be most fun and exciting to talk about.
I heard the CEO of Lets Encrypt, Warren Buffet, accidentally started a fire while charging his e-unicycle in the data centre and that knocked out the server that issues the certificates. They've got a backup, but it's in a safe only two people have keys to; one keyholder, Anne Hathaway, is at a parrot show in Singapore this week and her flight back is delayed due to fuel shortages. The other keyholder, Henry Kissinger, it turns out has been dead for 3 years.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#67How much of the internet is going to fail because of this?
It's an interesting thought experiment to consider how much of 'the internet' would still find a way to communicate with each other and fix the problem if somebody waved a magic wand and all http and https servers and clients magically disappeared worldwide instantly. For instance some of the folks who run core BGP at medium to large sized ISPs would revert back to a few legacy IRC channels and find each other to cha…
I bet we'd see a bunch of unexpected breakage in presumed-to-be-lower-level-than-http[s] infrastructure so that eg. your legacy IRC server goes down because it's running on rented hardware and the hosting provider's operations rely on some internal http services.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#68This is a compliance incident, we should be issuing again shortly. Update: Issuance is back up. Update: Preliminary incident report: https://bugzilla.mozilla.org/show_bug.cgi?id=2038351
> This is a compliance incident Uh. I don't know if I like the sound of that...
"compliance incident" is the catchall for everything from a spelling error on a CPS (certification practice statement) or being one second late on revocation, all the way up to to key compromise.
it is almost always closer to the spelling mistake side than it is the key compromise side of the spectrum.
a peak at https://bugzilla.mozilla.org/buglist.cgi?product=CA%20Progra... will show that most compliance issues, to the general public, are quite mundane.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#69Earlier quoted context omitted.
I just find it incredible that in 30+ years the industry hasn't adapted one bit to the brittle failure modes of certificates. I did some subcontract work with Verisign to deploy their CA infrastructure back in the early oughties and it felt like a solution was overdue way back then. I was at Google in the teensies when gmail broke due to expired SMTP certs. WAAAY overdue by then. Here we are, a decade later and it's…
Other than automating renewal - which we have made huge strides on - what adaption would you want to see?
I'd like to see better support for networks that aren't connected to the broader internet, or moving away from X.509. Note that these are contradictory. X.509 was intentionally designed to support offline verification and has a lot of elaborate ceremony to support it (like all the rest of the OSI stack). The industry just doesn't, so we get the worst of both worlds.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#70Earlier quoted context omitted.
Considering the open source nature of Letsencrypt, I wonder what the barriers/costs would be (theoretically) to a wealthy benefactor who wanted to duplicate its server side infrastructure and a core staffing level of persons, and fund a "parallel" equally trusted, alternative entity with a solid governing board. Same general idea how Acton funded the Signal foundation. Somewhere that none of the physical infrastructu…
Google has their own free ACME endpoint: https://pki.goog/
ZeroSSL should also be drop in