Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCP Outage

status.cloud.google.com

Post reply on HN
Print view View original

461–470 of 539 posts

Re: GCP Outage

#461

Post by londons_explore »

voytec wrote: post #388
terom wrote: post #371

Earlier quoted context omitted.

From the Cloudflare incident: > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable [...] Surprising, but not entirely unplausible for a GCP outage to spread to CF.

> outage of a 3rd party service that is a key dependency. Good to know that Cloudflare has services seemingly based on GCP with no redundancy.

Probably unintentional. "We just read this config from this URL at startup" can easily snowball into "if that URL is unavailable, this service will go down globally, and all running instances will fail to restart when the devops team try to do a pre-emptive rollback"

Re: GCP Outage

#462

Post by staplers »

the8472 wrote: post #351
SOLAR_FIELDS wrote: post #176

Earlier quoted context omitted.

There's no situation where the corporation controls the status page where you can trust the status page to have accurate information. None. The incentives will never be aligned in this regard. It's just too tempting and easy for the corp to control the narrative when they maintain their own status page. The only accurate status pages are provided by third party service checkers.

> The incentives will never be aligned in this regard. Well, yes, incentives, do big customers with wads of cash have an incentive to demand accurate reporting from their suppliers so they can react better rather than trying to identify issues? If there's systematic underreporting, then apparently not. Though in this case they did update their page.

  If there's systematic underreporting, then apparently not.
You answered your own question.

Re: GCP Outage

#463

Post by snowwrestler »

whatevertrevor wrote: post #433
derefr wrote: post #423

Earlier quoted context omitted.

Cloudflare isn't a cloud in the traditional sense; it's a CDN with extra smarts in the CDN nodes. CF's comparative advantage is in doing clever things with just-big-enough shared-nothing clusters deployed at every edge POP imaginable; not in building f-off huge clusters out in the middle of nowhere that can host half the Internet, including all their own services. As such, I wouldn't be overly surprised if all of CF'…

But then you do expose yourself to huge issues like this if your control plane is dependent on a single cloud provider, especially for a company that wants to be THE reverse proxy and CDN for the internet no?

Cloudflare does not actually want to reverse proxy and CDN the whole internet. Their business model is B2B; they make most of their revenue from a set of companies who buy at high price points and represent a tiny percentage of the total sites behind CF.

Scale is just a way to keep costs low. In addition to economies of scale, routing tons of traffic puts them in position to negotiate no-cost peering agreements with other bandwidth providers. Freemium scale is good marketing too.

So there is no strategic reason to avoid dependencies on Google or other clouds. If they can save costs that way, they will.

Re: GCP Outage

#464

Post by peanut-walrus »

johanyc wrote: post #458
peanut-walrus wrote: post #443

Earlier quoted context omitted.

Downdetector in incidents like this is 100% misinformation.

Why

Downdetector does not actually monitor the services. It aggregates user reports from socials etc. For large-scale incidents, the reports get really noisy and it will show that basically everything is down.

Re: GCP Outage

#465

Post by mbreese »

whatevertrevor wrote: post #433
derefr wrote: post #423

Earlier quoted context omitted.

Cloudflare isn't a cloud in the traditional sense; it's a CDN with extra smarts in the CDN nodes. CF's comparative advantage is in doing clever things with just-big-enough shared-nothing clusters deployed at every edge POP imaginable; not in building f-off huge clusters out in the middle of nowhere that can host half the Internet, including all their own services. As such, I wouldn't be overly surprised if all of CF'…

But then you do expose yourself to huge issues like this if your control plane is dependent on a single cloud provider, especially for a company that wants to be THE reverse proxy and CDN for the internet no?

True, but how often do outages like this happen? And when outages do happen, does Cloudflare have any more exposure than Google? I mean, if Google can’t handle it, why should Cloudflare be expected to? It also looks like the Cloudflare services have been somewhat restored, so whatever dependency there is looks like it’s able to be somewhat decoupled.

So long as the outages are rare, I don’t think there is much downside for Cloudflare to be tied to Google cloud. And if they can avoid the cost of a full cloud buildout (with multiple data centers and zones, etc…), even better.

Re: GCP Outage

#466

Post by SOLAR_FIELDS »

the8472 wrote: post #351
SOLAR_FIELDS wrote: post #176

Earlier quoted context omitted.

There's no situation where the corporation controls the status page where you can trust the status page to have accurate information. None. The incentives will never be aligned in this regard. It's just too tempting and easy for the corp to control the narrative when they maintain their own status page. The only accurate status pages are provided by third party service checkers.

> The incentives will never be aligned in this regard. Well, yes, incentives, do big customers with wads of cash have an incentive to demand accurate reporting from their suppliers so they can react better rather than trying to identify issues? If there's systematic underreporting, then apparently not. Though in this case they did update their page.

In practice how this plays out is that the big wads of cash holders will make demand, and Google (or whoever, Google is just the standin for the generic Corp here) will give them the actual information privately. It will still never be trusted to be reflected accurately on the public status page.

If you think about it from the corp’s perspective, it makes perfect sense. They weigh the risk reward. Are they going to be rewarded for the radical transparency or suffer fall out by acknowledging how bad of a dumpster fire the situation actually is? Easier for the corp to just lie, obscure and downplay to avoid having to even face that conundrum in the first place.

Re: GCP Outage

#468

Post by a2128 »

0xy wrote: post #426
ipsum2 wrote: post #180

Cloudflare is down too. From https://www.cloudflarestatus.com : Update - We are seeing a number of services suffer intermittent failures. We are continuing to investigate this and we will update this list as we assess the impact on a per-service level. Impacted services: Access WARP Durable Objects (SQLite backed Durable Objects only) Workers KV Realtime Workers AI Stream Parts of the Cloudflare dashboard Jun 12, 202…

Seems like a major wtf if Cloudflare is using GCP as a key dependency.

Some day Cloudflare will depend on GCP and GCP will depend on Cloudflare and AWS will rely on one of the two being online and Cloudflare will also depend on AWS and the internet will go down and no one will know how to restart it

Re: GCP Outage

#469

Post by rco8786 »

ashu1461 wrote: post #374
rco8786 wrote: post #350

Earlier quoted context omitted.

Google is 10 minutes late updating their status page. "So shady" It's really, really hard to make a status page realtime.

What makes you think it’s hard? We have AI generating songs and writing code, but setting up basic health checks is too much?

Yes. “Basic health checks” is not a real thing. I mean that genuinely.

> What makes you think it’s hard?

Being responsible (or rather, on a team of people responsible) for a status page of a big tech co made me think it’s hard.

“Is it down?” Is not a binary question.

Re: GCP Outage

#470

Post by whatevertrevor »

snowwrestler wrote: post #463
whatevertrevor wrote: post #433

Earlier quoted context omitted.

But then you do expose yourself to huge issues like this if your control plane is dependent on a single cloud provider, especially for a company that wants to be THE reverse proxy and CDN for the internet no?

Cloudflare does not actually want to reverse proxy and CDN the whole internet. Their business model is B2B; they make most of their revenue from a set of companies who buy at high price points and represent a tiny percentage of the total sites behind CF. Scale is just a way to keep costs low. In addition to economies of scale, routing tons of traffic puts them in position to negotiate no-cost peering agreements with…

Well I mean most of the internet in terms of traffic, not in terms of the corpus of sites. I agree the long-tail of websites is probably not profitable for them.
Post reply on HN