Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Public CA is down

status.pki.goog

Post reply on HN
Print view View original

101–110 of 166 posts

Re: Google Public CA is down

#101

Post by SchemaLoad »

moregrist wrote: post #99
PostOnce wrote: post #81

Earlier quoted context omitted.

They've begun injecting obnoxious ads into the downloadable mp3s on a lot of podcasts I've found. Hyperlocal ads for tire shops and bakeries. I don't want to buy tires, I want to learn about ______. The ads don't even make sense because they're irrelevant.

Welcome to radio 2.0. Give it another 10-20 years and your 2 hour podcasts will be 30 minutes of morning zoo DJ banter, 10 minutes of guests, and 1.5 hours of ads. We’ll have reached peak 90s all over again. With any luck we’ll avoid recreating the conditions for another Nickelback and can stay in the weird zone where Trip Hop and pop punk could chart at the same time.

And it will all be AI generated specifically for you live.

Re: Google Public CA is down

#102

Post by pvab3 »

tokyobreakfast wrote: post #30
bethekidyouwant wrote: post #27

Earlier quoted context omitted.

Never see these. Skill issue?

I'm inundated with them. YT has become borderline unusable. The homepage is nightmarish. Can't search for anything without being overwhelmed with shorts in the results, many unrelated to what I'm searching.

Just block them. I haven't seen a short in months.

Re: Google Public CA is down

#103

Post by SchemaLoad »

bawolff wrote: post #97
jtokoph wrote: post #24

> A fix to resolve the issue will roll out in about 8 hours oof

In theory 8 hours of downtime should be fine for a CA. Obviously not ideal, but the pki system is not meant to be a live system.

Fairly sure it used to be pretty much a manual process where someone had to actually process your request for a certificate on the other side.

Re: Google Public CA is down

#104

Post by antonvs »

OhMeadhbh wrote: post #66

I worked at RSADSI when I was a kid and supported the custom spin of TIPEM Hayden and Sophia used at Verisign. This brings back some very bad memories. But... hopefully... people created overlapping windows of cert validity so there's always a valid cert available for their services and can tolerate the CA being out of action for 8(?) hours. Imagine if your TGS/Kerberos or AWS IAM IdP was down for 8 hours.

For persistent services using the affected ACME API, the window is usually 30 days.

But that didn’t stop Youtube and Youtube TV from going down hard. I imagine they’re provisioning ephemeral VMs or service instances and relying on them being able to get certs immediately, or something like that.

Re: Google Public CA is down

#105

Post by SchemaLoad »

JyB wrote: post #88
bigbuppo wrote: post #45

Earlier quoted context omitted.

It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data. It will forever be known as The Great Oops.

I don’t know if you’re being serious but that’s laughable

The idea that all customer data will be deleted is far fetched, but I feel like there have been some massive incidents. Crowdstrike comes to mind, but I feel its entirely possible that Apple/Google/etc could push out some kind of config update which bricks phones in a way they are unable to download another update to fix them.

Though I'm sure the major players are all over this risk which is why it hasn't happened.

Re: Google Public CA is down

#106

Post by Arainach »

ocdtrekkie wrote: post #95
Arainach wrote: post #92

Earlier quoted context omitted.

.....no? "We deployed this private cloud with a missing parameter and it wasn't caught" is as different from "we wiped out all customer data" as hello world is from Kubernetes. No one promised this "should be impossible". Did you confuse "we'll take steps to ensure this never happens again"?

It's pretty much half the puzzle actually. You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner. The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes…

The release process, monitoring checks, etc. for a customer's private cloud is generally significantly different from the release process for a global product. I'm not going to get any more specific for all the standard NDA reasons, but having worked for Google and Microsoft among others....no, the risk you describe doesn't translate from one to the other.

Re: Google Public CA is down

#107

Post by antonvs »

jsheard wrote: post #62
tokyobreakfast wrote: post #60

Earlier quoted context omitted.

How many servers are set up with CA redundancy? I've yet to see one let alone hear of this practice.

For one, Cloudflare uses four different CAs almost interchangeably. Caddy also makes it easy to configure ACME failover if you're self-hosting, and defaults to using two different CAs if you don't specify any. Frankly even with no CA redundancy, downtime would have to drag on for weeks to actually disrupt renewals. ACME certs usually get rotated after about 2/3rds of their duration has expired, so the upcoming 45 day…

So the question is why this hit Youtube and Youtube TV so hard. Presumably they’re relying on ephemeral instances being able to get certs immediately, or something like that.

(Or an unrelated failure, of course)

Re: Google Public CA is down

#109

Post by ocdtrekkie »

Arainach wrote: post #106
ocdtrekkie wrote: post #95

Earlier quoted context omitted.

It's pretty much half the puzzle actually. You contend there's no global rm rf for a global cloud provider, but clearly a missing parameter can rm rf a customer in an irrecoverable manner. The only half you're missing is... how every major cloud outage happens today... a bad configuration update. These companies have hundreds of thousands of servers, but they also use orchestration tools to distribute sets of changes…

The release process, monitoring checks, etc. for a customer's private cloud is generally significantly different from the release process for a global product. I'm not going to get any more specific for all the standard NDA reasons, but having worked for Google and Microsoft among others....no, the risk you describe doesn't translate from one to the other.

I understand you believe the checks cannot fail that catastrophically, and I agree that the likelihood they do is quite low.

But it can happen, and it only has to happen once. (Also FYI, telling me your work history just tells me you've drunk the koolaid, ain't proof you know more.)

Re: Google Public CA is down

#110

Post by rossjudson »

Arainach wrote: post #64
bigbuppo wrote: post #45

Earlier quoted context omitted.

It's inevitable that one of the major cloud providers will irrecoverably delete all customer data with one single fat-fingered command. Though in google's case I'll also consider the prophecy to be fulfilled if they delete their own data. It will forever be known as The Great Oops.

It's not inevitable, it's essentially impossible. There are a few things that can cause tremendously widespread outages , essentially all of them network configuration changes. Actually deleting customer data is dramatically more difficult to the point of impossible - there are so many different services in so many different locations with so many layers of access control. There is no "one command" that can do such a…

Delete a decryption key. Good luck! I'll see you at the end of time.

Break your control plane, and you can't stop the propagation of poison.

Propagate the wrong trust bundle... everywhere.

Also, it's not about the delete command. It's about the automatic cleanup following behind it that shreds everything, or repurposes the storage.

Post reply on HN