Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

letsencrypt.status.io

Post reply on HN
Print view View original

51–60 of 97 posts

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#51

Post by dochtman »

walrus01 wrote: post #13
mark_round wrote: post #5

That's really not good. Fortunately I'm not using any short-lived certificates like the recently announced 6 day certs, so have some breathing room. Without further details, I'd imagine anyone with a short-lived cert is getting a bit sweaty right now. Let's Encrypt has become one of those pieces of critical Internet infrastructure that just quietly hums away in the background, the fact that they've stopped ALL issuan…

Considering the open source nature of Letsencrypt, I wonder what the barriers/costs would be (theoretically) to a wealthy benefactor who wanted to duplicate its server side infrastructure and a core staffing level of persons, and fund a "parallel" equally trusted, alternative entity with a solid governing board. Same general idea how Acton funded the Signal foundation. Somewhere that none of the physical infrastructu…

A lot of Let’s Encrypt is not the software but a bunch of auditing and process that ensure compliance and make it legible to the required auditors.

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#52

Post by eqvinox »

walrus01 wrote: post #46
gabeio wrote: post #28

Earlier quoted context omitted.

> This is a compliance incident Uh. I don't know if I like the sound of that...

Indeed. "Compliance" can mean some internal audit/monitoring system has tripped and requires in depth investigation and preservation of logging, or it can mean "federal law enforcement with badges are right now standing in our datacenter and/or NOC serving a court order".

Federal law enforcement in your DC isn't something you'd call a "compliance" issue, that's not what that term means. Yes it's various derivatives of the English word "comply", but this is a field of well-defined verbiage, and that ain't it. Compliance means they failed (or are being questioned) about following particular practices that they have agreed to, nothing else really.

NB: "legal compliance" is another term. So is "{legal,lawful} enforcement"

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#53

Post by jaas »

mark_round wrote: post #5

That's really not good. Fortunately I'm not using any short-lived certificates like the recently announced 6 day certs, so have some breathing room. Without further details, I'd imagine anyone with a short-lived cert is getting a bit sweaty right now. Let's Encrypt has become one of those pieces of critical Internet infrastructure that just quietly hums away in the background, the fact that they've stopped ALL issuan…

Stopping all issuance is an pretty standard response if a CA thinks what they are issuing might be non-compliant in any way. It's an action we're required to take. It's not necessarily a sign of a more dramatic failure mode or key compromise. That said, the impact is the same for as long as the downtime lasts so it is unfortunate and we're sorry for the disruption.

I don't think the premise behind short lived (six day) certificates being viable is that CA issuance never goes down. Sure, the runway is shorter, but not that short. Most down time is a few hours or less, which is not a problem for six day certificates that should be renewed every three days.

Short lived certificates are optional though, so if it's not worth it to you there are longer lifetime options.

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#54

Post by walrus01 »

t1234s wrote: post #27

How much of the internet is going to fail because of this?

It's an interesting thought experiment to consider how much of 'the internet' would still find a way to communicate with each other and fix the problem if somebody waved a magic wand and all http and https servers and clients magically disappeared worldwide instantly.

For instance some of the folks who run core BGP at medium to large sized ISPs would revert back to a few legacy IRC channels and find each other to chat and figure out WTF is going on.

"the internet" would still exist, a subset of the application layer stuff that runs on top it wouldn't...

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#55

Post by reaperducer »

aroman wrote: post #17
cedws wrote: post #7

Discord is out too right now, probably unrelated though.

Just speculating, but I don't think it's unrelated. Discord heavily utilizes Cloudflare, and Cloudflare uses Let's Encrypt for a certificate issuance. If they happened to have a certificate signing dependency in some operational rollout today, I think it could explain it. Certainly the timing is very correlated.

Just speculating

Then why post? HN is for informed discussion, not every random thought in someone's head.

Certainly the timing is very correlated.

I had chocolate ice cream for breakfast. Certainly the timing is very corrolated [sic].

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#56

Post by nottorp »

mark_round wrote: post #5

That's really not good. Fortunately I'm not using any short-lived certificates like the recently announced 6 day certs, so have some breathing room. Without further details, I'd imagine anyone with a short-lived cert is getting a bit sweaty right now. Let's Encrypt has become one of those pieces of critical Internet infrastructure that just quietly hums away in the background, the fact that they've stopped ALL issuan…

> like the recently announced 6 day certs

Just you wait for the 1 hour and 59 minutes certs! For security!

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#57

Post by jaas »

walrus01 wrote: post #49
tptacek wrote: post #48

Earlier quoted context omitted.

At times like this it's worth remembering that message boards strongly favor whatever narrative is going to be most fun and exciting to talk about.

I sincerely hope it's the most mundane and least spectacular explanation possible, just saying from my point above that compliance has a very wide range of possible meanings and interpretations (also depending on the background/career POV of the reader), until the incident is further explained..

In that sense, prepare yourself to be bored.

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#58

Post by nottorp »

striking wrote: post #33
eqvinox wrote: post #16

Earlier quoted context omitted.

Short-lived = 6 days. Even if you reissue after 2 or 3 days, that's… not a lot of breathing room.

You have to opt in, and they are honest about the tradeoffs when discussing them: > Short-lived certificates are opt-in and we have no plan to make them the default at this time. Subscribers that have fully automated their renewal process should be able to switch to short-lived certificates easily if they wish, but we understand that not everyone is in that position and generally comfortable with this significantly s…

> no plan to make them the default at this time

At this time! Boil the frog slowly...

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#59

Post by walrus01 »

dochtman wrote: post #51
walrus01 wrote: post #13

Earlier quoted context omitted.

Considering the open source nature of Letsencrypt, I wonder what the barriers/costs would be (theoretically) to a wealthy benefactor who wanted to duplicate its server side infrastructure and a core staffing level of persons, and fund a "parallel" equally trusted, alternative entity with a solid governing board. Same general idea how Acton funded the Signal foundation. Somewhere that none of the physical infrastructu…

A lot of Let’s Encrypt is not the software but a bunch of auditing and process that ensure compliance and make it legible to the required auditors.

I understand there's probably a big thorny problem of duplicating the corporate process/policies on the human level that ensure compliance, but is the back-end software pipelining stuff to CT logs not also something that can be replicated? Or is it not part of the server side stuff which has been open sourced?

https://letsencrypt.org/docs/ct-logs/

Re: Let’s Encrypt: Stopping Issuance for Potential Incident – Resolved

#60

Post by nicolas_17 »

rvnx wrote: post #37
tptacek wrote: post #24

Earlier quoted context omitted.

Josh Aas is on the thread. It's a compliance issue, they expect to be issuing shortly.

What if they get kicked out of trusted roots because non-compliant ?

That's why they take incidents like this seriously and stop issuance until it's fixed. They could get kicked out of trusted roots otherwise.
Post reply on HN