Earlier quoted context omitted.
Josh Aas is on the thread. It's a compliance issue, they expect to be issuing shortly.
What if they get kicked out of trusted roots because non-compliant ?
Change your config to ZeroSSL or another free ACME provider?
Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
71–80 of 97 posts
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#72Earlier quoted context omitted.
Just speculating, but I don't think it's unrelated. Discord heavily utilizes Cloudflare, and Cloudflare uses Let's Encrypt for a certificate issuance. If they happened to have a certificate signing dependency in some operational rollout today, I think it could explain it. Certainly the timing is very correlated.
On my account they always serve Google issued certificates. There is also Let’s encrypt certificate but it is not used though. I guess that’s a fail-safe.
In Cloudflare Enterprise you can pick either or leave it on auto. Iirc there's a 3rd option but I don't know if it's still supported (Terraform and SDKs used to have it in the enum)
https://developers.cloudflare.com/ssl/reference/certificate-...
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#73Earlier quoted context omitted.
It's an interesting thought experiment to consider how much of 'the internet' would still find a way to communicate with each other and fix the problem if somebody waved a magic wand and all http and https servers and clients magically disappeared worldwide instantly. For instance some of the folks who run core BGP at medium to large sized ISPs would revert back to a few legacy IRC channels and find each other to cha…
I bet we'd see a bunch of unexpected breakage in presumed-to-be-lower-level-than-http[s] infrastructure so that eg. your legacy IRC server goes down because it's running on rented hardware and the hosting provider's operations rely on some internal http services.
This is extremely likely in the case of many automated provisioning, billing, and web interface control panel systems for shared hosting platforms, VPS, virtual machine service providers that likely do something https to https internally to communicate between tooling.
In my intentionally absurd theoretical scenario, what would remain up would be the bare metal in colocation in certain service providers' environments...
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#74Earlier quoted context omitted.
You have to opt in, and they are honest about the tradeoffs when discussing them: > Short-lived certificates are opt-in and we have no plan to make them the default at this time. Subscribers that have fully automated their renewal process should be able to switch to short-lived certificates easily if they wish, but we understand that not everyone is in that position and generally comfortable with this significantly s…
> no plan to make them the default at this time At this time! Boil the frog slowly...
Is the frog the guy that still won't automate their certificates?
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#75Hopefully it's just a technical issue and not something like a key compromise. This could have disastrous effects considering how much of the web runs on LE certs these days. Granted if it's configured properly everyone should have 30 days of leeway before having to issue new certs...
"We have been made aware of a potential incident and are shutting down all issuance" seems to lean towards the latter and not simply a technical issue :(
What makes you think that?
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#76Earlier quoted context omitted.
Josh Aas is on the thread. It's a compliance issue, they expect to be issuing shortly.
What if they get kicked out of trusted roots because non-compliant ?
You don't get kicked out of trusted roots for non-compliance, you get kicked out for continuing to knowingly issue non-compliant certs, failing to revoke non-compliant certs in a timely fashion once discovered, etc.
Pausing issuance immediately upon discovery of a compliance issue is the absolute correct response so as long as they do their followup appropriately there is absolutely zero risk of being distrusted.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#77Earlier quoted context omitted.
Indeed. "Compliance" can mean some internal audit/monitoring system has tripped and requires in depth investigation and preservation of logging, or it can mean "federal law enforcement with badges are right now standing in our datacenter and/or NOC serving a court order".
Federal law enforcement in your DC isn't something you'd call a "compliance" issue, that's not what that term means. Yes it's various derivatives of the English word "comply", but this is a field of well-defined verbiage, and that ain't it . Compliance means they failed (or are being questioned) about following particular practices that they have agreed to, nothing else really. NB: "legal compliance" is another term.…
Compliance here means compliance with the CA/B Forum Baseline Requirements (and similar other policies), which cover a lot of operational obligations, from character encoding to physical security.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#78Earlier quoted context omitted.
I just find it incredible that in 30+ years the industry hasn't adapted one bit to the brittle failure modes of certificates. I did some subcontract work with Verisign to deploy their CA infrastructure back in the early oughties and it felt like a solution was overdue way back then. I was at Google in the teensies when gmail broke due to expired SMTP certs. WAAAY overdue by then. Here we are, a decade later and it's…
Other than automating renewal - which we have made huge strides on - what adaption would you want to see?
The number one thing for me would be to standardize methods to implement soft failures. Minimally in standard clients and libraries the ability to warn when certs are nearing expiration. Cert extensions to declare lifecycle expectations and possibly even warning endpoints for notification. Basically some way to empirically look at a valid cert and know something is wrong before it fails.
There are all sorts of potential privacy/security issues with any feature built in this area so it would have to be done carefully, but I think useful improvements could easily be made.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#79Earlier quoted context omitted.
What if they get kicked out of trusted roots because non-compliant ?
You don't get kicked out of trusted roots for non-compliance, you get kicked out for continuing to knowingly issue non-compliant certs, failing to revoke non-compliant certs in a timely fashion once discovered, etc. Pausing issuance immediately upon discovery of a compliance issue is the absolute correct response so as long as they do their followup appropriately there is absolutely zero risk of being distrusted.
> You don't get kicked out of trusted roots for non-compliance
Of course you do, it's the main reason CAs fix compliance issues so fast.
Symantec, WoSign, Entrust, etc repeatedly had non-compliance issues and that led to them being removed (even if fixed)
Here was not a big issue: they forgot a flag to narrow the delegation of trust (but nobody knew that a few hours ago)
Still it can be very problematic, there is a quite similar situation here https://bugzilla.mozilla.org/show_bug.cgi?id=1883843
A basic non-compliance issue, just a web link missing, but huge consequences if they don’t fix it.
Repeated non-compliance (like the Symantec) will eventually get you removed even if fixed.
The core definition of losing “trust” in someone.
Keep in mind that few hours ago, nobody knew what the violation was. Turns out it was an easy fix.
Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved
#80Earlier quoted context omitted.
You don't get kicked out of trusted roots for non-compliance, you get kicked out for continuing to knowingly issue non-compliant certs, failing to revoke non-compliant certs in a timely fashion once discovered, etc. Pausing issuance immediately upon discovery of a compliance issue is the absolute correct response so as long as they do their followup appropriately there is absolutely zero risk of being distrusted.
> You don't get kicked out of trusted roots for non-compliance Of course you do, it's the main reason CAs fix compliance issues so fast. Symantec, WoSign, Entrust, etc repeatedly had non-compliance issues and that led to them being removed (even if fixed) Here was not a big issue: they forgot a flag to narrow the delegation of trust (but nobody knew that a few hours ago) Still it can be very problematic, there is a q…
You didn't actually respond to what the preceding comment argued. They were just pointing out the distinction between Symantec and WoSign and ordinary compliance events.