Google Public CA is down • Hacker News Board

Re: Google Public CA is down

#161

Post by [deleted] » Wed, Feb 18, 2026, 6:28 PM UTC

[deleted]

Re: Google Public CA is down

#162

Post by dilyevsky » Wed, Feb 18, 2026, 6:50 PM UTC

rconti wrote: post #131

> The fix has been rolled out and the issuance flow has been undrained. We again apologize for the inconvenience. issuance flow has been undrained?

google sre speak

Re: Google Public CA is down

#163

Post by DANmode » Thu, Feb 19, 2026, 3:00 AM UTC

Arainach wrote: post #127

bigbuppo wrote: post #116

Earlier quoted context omitted.

Do you not remember crowdstrike?

Again: an outage caused by a config change is different from data loss. The remediation was painful but it was not data loss.

What if a machine was supposed to be running to capture data?

Re: Google Public CA is down

#164

Post by goto1 » Thu, Feb 19, 2026, 4:06 AM UTC

agwa wrote: post #76

bathtub365 wrote: post #53

The status history on the page makes it seem like this was intentional? > 17 Feb 2026 11:32 PST A rollout is going to prevent issuance from occurring. We will provide an estimate on when issuance will stop. > 17 Feb 2026 12:14 PST Issuance is beginning to stop. A fix to resolve the issue will roll out in about 8 hours

This usually indicates that the CA was issuing non-compliant certificates and needed to prevent further non-compliance. Will be interesting to watch Bugzilla for the incident report: https://bugzilla.mozilla.org/buglist.cgi?product=CA%20Progra...

https://bugzilla.mozilla.org/show_bug.cgi?id=2017747

Re: Google Public CA is down

#165

Post by jamiemallers » Thu, Feb 19, 2026, 9:09 AM UTC

bigbuppo wrote: post #121

ocdtrekkie wrote: post #95

Earlier quoted context omitted.

It's pretty much half the puzzle actually. You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner. The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes…

And the most telling thing about most of these outages is that the provider later admits in their postmortem that they just didn't really understand how the system they made worked until it fell over and were forced to learn how it really works. It's the sort of thing that used to keep me up at night.

[dead]

Re: Google Public CA is down

#166

Post by SchemaLoad » Thu, Feb 19, 2026, 10:35 PM UTC

Ayesh wrote: post #142

SchemaLoad wrote: post #103

Earlier quoted context omitted.

Fairly sure it used to be pretty much a manual process where someone had to actually process your request for a certificate on the other side.

Yes, and it's not that long ago, or I aged really quickly. For code signing certificates and EV certificates, (and OV certificates, if they are even alive), this is still the case.

It's been 11 years now since Lets Encrypt started with automated certs. EV certs I think died a long time ago.