GCP Outage • Hacker News Board

Re: GCP Outage

#231

Post by cwillu » Thu, Jun 12, 2025, 6:59 PM UTC

toomuchtodo wrote: post #85

hambro wrote: post #11

@dang could you merge this and https://news.ycombinator.com/item?id=44260669 ?

No notifications for mentions, have to email the mods at the hn@ email address.

Do we know if email is still working? kidding-but-not-really-because-gmail…

Re: GCP Outage

#232

Post by cyberpunk » Thu, Jun 12, 2025, 7:00 PM UTC

ransom1538 wrote: post #171

carter-0 wrote: post #132

Earlier quoted context omitted.

[dead]

It's the entire internet. Check oracle cloud, etc etc. The ENTIRE INTERNET.

Quick! Pirate as much music as possible before it goes for good! ;)

Re: GCP Outage

#233

Post by whatevertrevor » Thu, Jun 12, 2025, 7:00 PM UTC

rvnx wrote: post #187

VWWHFSfQ wrote: post #170

Earlier quoted context omitted.

There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.

Absolutely possible. Though there is something curious: https://www.cloudflarestatus.com/ At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.". So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.

Doesn't cloudflare have its own infrastructure, it's wild to me that both these things are down presumably together with this size of a blast radius.

Re: GCP Outage

#234

Post by DonHopkins » Thu, Jun 12, 2025, 7:02 PM UTC

Damn you Bart Simpson!

https://en.wikipedia.org/wiki/Bart_Gets_Famous

Re: GCP Outage

#235

Post by tonyhart7 » Thu, Jun 12, 2025, 7:02 PM UTC

carter-0 wrote: post #132

thanhhaimai wrote: post #2

The status page is green, but there are outages reported: https://downdetector.com/status/google-cloud/

[dead]

wtf is going on

Re: GCP Outage

#236

Post by tonyhart7 » Thu, Jun 12, 2025, 7:02 PM UTC

ransom1538 wrote: post #171

carter-0 wrote: post #132

Earlier quoted context omitted.

[dead]

It's the entire internet. Check oracle cloud, etc etc. The ENTIRE INTERNET.

are there nuclear war or something???

Re: GCP Outage

#237

Post by dinvlad » Thu, Jun 12, 2025, 7:03 PM UTC

#HugOps

Re: GCP Outage

#238

Post by devmor » Thu, Jun 12, 2025, 7:03 PM UTC

Incident report published: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...

Re: GCP Outage

#239

Post by james_m1231 » Thu, Jun 12, 2025, 7:03 PM UTC

My firebase hosting and firestore db are back online, but GCP console and Google SQL instances are still having serious issues as of 7:00pm UTC.

Re: GCP Outage

#240

Post by mpalmer » Thu, Jun 12, 2025, 7:04 PM UTC

sss111 wrote: post #204

kjuulh wrote: post #169

Earlier quoted context omitted.

Something must be preventing them updating the status page at this point. Of course they could still deem it not enough, but just from my limited tests, docker, buf, etc (it may not be GCP that is down, but it is quite the coincidence). are outright down. I'd wager that this is much more widespread.

I'm actually on a bridge call with Google Cloud, we're a large customer -- I just learned today that their status page is not automated, instead someone actually manually updates it!

The bigger you are, the more you want a human involved in the decision to publicly declare an incident.