Live data from Hacker News

Google Public CA is down

status.pki.goog

141–150 of 166 posts

Re: Google Public CA is down

#141
post #83
post #18

Everyone loves to say they work at $FAMOUS_COMPANY, but when something like this happens, no-one will say that they did this. Looking forward to the post-mortem.

Oh I am more than happy to tell people how I took down entire Google Cloud 11 years ago. I mean, of course to the level of details Google is comfortable with to share externally :)

I'll bite; ok, what'd you do? :)

Re: Google Public CA is down

#142
post #97

Earlier quoted context omitted.

In theory 8 hours of downtime should be fine for a CA. Obviously not ideal, but the pki system is not meant to be a live system.

Fairly sure it used to be pretty much a manual process where someone had to actually process your request for a certificate on the other side.

Yes, and it's not that long ago, or I aged really quickly.

For code signing certificates and EV certificates, (and OV certificates, if they are even alive), this is still the case.

Re: Google Public CA is down

#143

Earlier quoted context omitted.

Delete a decryption key. Good luck! I'll see you at the end of time. Break your control plane, and you can't stop the propagation of poison. Propagate the wrong trust bundle... everywhere. Also, it's not about the delete command. It's about the automatic cleanup following behind it that shreds everything, or repurposes the storage.

Children of the kubernetic line.

Cyclic infrastructure dependencies suck :(

Re: Google Public CA is down

#144
The CA outage is hitting a lot of services, but yeah, Heroku's been on a slow decline since the Salesforce acquisition. Free tier killed, pricing creep, stagnant innovation. Even when it's not their fault, you start wondering if it's worth the risk of being on a platform that feels like it's in maintenance mode.

Re: Google Public CA is down

#145
Yeah, if Heroku's cert rotation depends on Google's CA and it tried to renew during the outage window, that'd definitely cause problems. The 8-hour ETA is rough. This is why multi-CA fallback configs exist, but most platforms don't bother until they get burned by something like this. Worth checking if your apps are actually affected or if it's just the dashboard/API having issues.

Re: Google Public CA is down

#146
post #84

Earlier quoted context omitted.

What qualifies as a non-compliant certificate?

It doesn't comply with one or more root store policies (which all incorporate the Baseline Requirements by reference, which incorporate various specs, such as RFC5280, by reference). Mozilla root store policy: https://www.mozilla.org/en-US/about/governance/policies/secu... Chrome root store policy: https://googlechrome.github.io/chromerootprogram/ Apple root store policy: https://www.apple.com/certificateauthority/ca…

Something is badly borked when the protections against an imaginary problem cause a real problem.

Re: Google Public CA is down

#147

People went ballistic on me a few months ago for bringing this up, but this is exactly the kind of outage that makes me really, really worried about extremely short lived certificates. https://news.ycombinator.com/item?id=46118371

Your license to website has been revoked.

You're joking, but still: that's one very possible outcome of both requiring centrally issued certificates for security reasons and browsers refusing to display websites without.

Effectively certificates are now a license to publish.

Re: Google Public CA is down

#148
post #84

Earlier quoted context omitted.

It doesn't comply with one or more root store policies (which all incorporate the Baseline Requirements by reference, which incorporate various specs, such as RFC5280, by reference). Mozilla root store policy: https://www.mozilla.org/en-US/about/governance/policies/secu... Chrome root store policy: https://googlechrome.github.io/chromerootprogram/ Apple root store policy: https://www.apple.com/certificateauthority/ca…

Something is badly borked when the protections against an imaginary problem cause a real problem.

Baseline requirements are not an imaginary problem. All of them have a legitimate reason for existing. You could argue that some "are not that big of a deal", but that's exactly the point, the overbearing and overly specific requirements serve both their own purpose and double as Van Halen's "no brown M&Ms" clause: if the CA screws them up, either by malice or incompetence and doesn't immediately catch them and self-report, then you know they have no way of telling what other things they are screwing up. And if you're in the business of selling trust, that instantly makes you untrustworthy.

There are countless Bugzilla reports of clearly unprofessional CAs trying to get away with doing whatever they want, get caught, say "it's no big deal", fail to learn the lesson and eventually get kicked out, much to the chagrin and bewilderment of their management, irate that some nerds on the Internet could ruin their business, failing to understand that following the scripture of the Internet nerds is the #1 requirement of the business they chose to run.

Re: Google Public CA is down

#150

Earlier quoted context omitted.

Your license to website has been revoked.

You're joking, but still: that's one very possible outcome of both requiring centrally issued certificates for security reasons and browsers refusing to display websites without. Effectively certificates are now a license to publish.

On a PC we atleast have an out.

On mobile, user certs are pretty much ignored unless opted in by apps. Even firefox allows user certs (for now) but only via an obscure hidden config.

This means we cannot use self-hosted services even using a VPN with official apps without getting a signed cert.

Post reply on HN