GCP Outage • Hacker News Board

Re: GCP Outage

#381

Post by lysace » Thu, Jun 12, 2025, 8:13 PM UTC

brentm wrote: post #358

throitallaway wrote: post #283

Earlier quoted context omitted.

I believe Downdetector displays user reports.

Yea I am pretty sure that if you're checking if a service is down your essentially casting a vote that indicates that service is down.

Kind of a missed opportunity for Ookla - who's running both downdetector.com and speedtest.net.

The have software running in most ISPs around the world:

https://help.speedtest.net/hc/en-us/articles/360039164793-Ho...

(OTOH, it's not always trivial to define/detect an outage.)

Re: GCP Outage

#382

Post by augbog » Thu, Jun 12, 2025, 8:14 PM UTC

Cloudflare Outage also just updated

> Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information

Re: GCP Outage

#383

Post by deepsun » Thu, Jun 12, 2025, 8:15 PM UTC

ransom1538 wrote: post #171

carter-0 wrote: post #132

Earlier quoted context omitted.

[dead]

It's the entire internet. Check oracle cloud, etc etc. The ENTIRE INTERNET.

Hacker News is fine.

Re: GCP Outage

#384

Post by jug » Thu, Jun 12, 2025, 8:16 PM UTC

ashu1461 wrote: post #374

rco8786 wrote: post #350

Earlier quoted context omitted.

Google is 10 minutes late updating their status page. "So shady" It's really, really hard to make a status page realtime.

What makes you think it’s hard? We have AI generating songs and writing code, but setting up basic health checks is too much?

An AI generated status page would be the epitome of 2025.

Re: GCP Outage

#385

Post by voytec » Thu, Jun 12, 2025, 8:16 PM UTC

0xffany wrote: post #145

Everything appears to be down as of 18:43 UTC... https://downdetector.com/

Yeah. This service was presenting charts likely probed from inside GCP. I was on a call with a Google rep, someone pointed out that "AWS is also down" and I foolishly said something about "possible BGP attack" out of spite, before checking AWS availability myself. Shame on me.

Re: GCP Outage

#386

Post by whalesalad » Thu, Jun 12, 2025, 8:17 PM UTC

mrngm wrote: post #373

0xffany wrote: post #184

Does anyone know of a good dashboard to check for such BGP routing anomalies as (apparently) this one? I am currently digging around https://radar.cloudflare.com/routing but it doesn't show which routes were actually leaked. I would love if anyone has any good tool recommendations!

My default go-to: https://bgp.tools/ Why would you think this outage is (internet) BGP related?

Cloudflare runs all their own bare metal servers. Seems odd that they would be impacted by Google cloud. Same can be said for all the other issues on downdetector. This points to a broad issue at the core internet which could certainly be related to BGP.

Re: GCP Outage

#387

Post by thefourthchime » Thu, Jun 12, 2025, 8:18 PM UTC

cryptonector wrote: post #353

atonse wrote: post #26

Getting a lot of errors for Claude Sonnet 4 (Cursor) and Gemini Pro. Nooooo I'm going to have to use my brain again and write 100% of my code like a caveman from December 2024.

Devs before June 12, 2025: "Ai? Pfft, hallucination central. They'll never replace me!" Devs during June 12, 2025 GCP outage: "What, no AI?! Do you think I'm a slave?!"

So true

Re: GCP Outage

#388

Post by voytec » Thu, Jun 12, 2025, 8:18 PM UTC

terom wrote: post #371

rvnx wrote: post #187

Earlier quoted context omitted.

Absolutely possible. Though there is something curious: https://www.cloudflarestatus.com/ At Cloudflare it started with: "Investigating - Cloudflare engineering is investigating an issue causing Access authentication to fail.". So this would somehow validate the theory of auth/quotas started failing right after Google, but what happened after ?! Pure snowballing ? That sounds a bit crazy.

From the Cloudflare incident: > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable [...] Surprising, but not entirely unplausible for a GCP outage to spread to CF.

> outage of a 3rd party service that is a key dependency.

Good to know that Cloudflare has services seemingly based on GCP with no redundancy.

Re: GCP Outage

#389

Post by itdependsnet » Thu, Jun 12, 2025, 8:20 PM UTC

Any chance this is the root being that so many different services are effected? https://github.com/kubernetes/kops/issues/17433

Re: GCP Outage

#390

Post by sebzim4500 » Thu, Jun 12, 2025, 8:21 PM UTC

leoh wrote: post #307

If Google Chat is down per https://www.google.com/appsstatus/dashboard/ , the ability for Google engineers to communicate among themselves impaired, despite SREs having IRC as a backup.

TIL Google chat hasn't been killed yet