Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCP Outage

status.cloud.google.com

Post reply on HN
Print view View original

471–480 of 539 posts

Re: GCP Outage

#471

Post by tibbar »

voytec wrote: post #388
terom wrote: post #371

Earlier quoted context omitted.

From the Cloudflare incident: > Cloudflare’s critical Workers KV service went offline due to an outage of a 3rd party service that is a key dependency. As a result, certain Cloudflare products that rely on KV service to store and disseminate information are unavailable [...] Surprising, but not entirely unplausible for a GCP outage to spread to CF.

> outage of a 3rd party service that is a key dependency. Good to know that Cloudflare has services seemingly based on GCP with no redundancy.

What's the alternative here? Do you want them to replicate their infrastructure across different cloud providers with automatic fail-over? That sounds -- heck -- I don't know if modern devops is really up to that. It would probably cause more problems than it would solve...

Re: GCP Outage

#473

Post by gcpman »

VWWHFSfQ wrote: post #170
rvnx wrote: post #140

It looks like that it is a central service @ Google called Chemist that is down. "Chemist checks the project status, activation status, abuse status, billing status, service status, location restrictions, VPC Service Controls, SuperQuota, and other policies." -> This would totally explain the error messages "visibility check (of the API) failed" and "cannot load policy" and the wide amount of services affected. cf. h…

There are multiple internet services down, not just GCP. It's just possible that this "Chemist" service is especially externally affected which is why the failures are propagating to the their internal GCP network services.

perhaps the person who maintains Chemist took the buyout

https://www.businessinsider.com/google-return-office-buyouts...

Re: GCP Outage

#474

Post by arccy »

tibbar wrote: post #471
voytec wrote: post #388

Earlier quoted context omitted.

> outage of a 3rd party service that is a key dependency. Good to know that Cloudflare has services seemingly based on GCP with no redundancy.

What's the alternative here? Do you want them to replicate their infrastructure across different cloud providers with automatic fail-over? That sounds -- heck -- I don't know if modern devops is really up to that. It would probably cause more problems than it would solve...

They're a company that has to run their own datacenters, you'd expect them to not fall over when a public cloud does.

Re: GCP Outage

#475

Post by arccy »

derefr wrote: post #423
whatevertrevor wrote: post #233

Earlier quoted context omitted.

Doesn't cloudflare have its own infrastructure, it's wild to me that both these things are down presumably together with this size of a blast radius.

Cloudflare isn't a cloud in the traditional sense; it's a CDN with extra smarts in the CDN nodes. CF's comparative advantage is in doing clever things with just-big-enough shared-nothing clusters deployed at every edge POP imaginable; not in building f-off huge clusters out in the middle of nowhere that can host half the Internet, including all their own services. As such, I wouldn't be overly surprised if all of CF'…

They're pushing workers more as a compute platform

Plus their past outage reports indicate they should be running their own DC: https://blog.cloudflare.com/major-data-center-power-failure-...

Re: GCP Outage

#476

Post by ManBeardPc »

The cloud enables you to scale. It allows us to distribute systems across multiple regions and data centers. Seems that this is true for outages as well.

The PHP application I wrote as a student running on a single self-hosted server had a higher uptime than any of the cloud providers or redundant system I have seen so far. If you don’t need the cloud for scalability, do it yourself and save yourself the trouble and money. Most companies would be better off investing into some IT staff instead of giving away their systems in the hands of some proprietary and insanely complex cloud environment. You are becoming dependent on someone you don’t know, have no control over and can’t talk with directly. Also the single point of failure is just shifting: from your system to whatever system is managing the cloud. Guess one advantage is that you can shift the blame to someone else…

Re: GCP Outage

#477

Post by lysace »

a2128 wrote: post #468
0xy wrote: post #426

Earlier quoted context omitted.

Seems like a major wtf if Cloudflare is using GCP as a key dependency.

Some day Cloudflare will depend on GCP and GCP will depend on Cloudflare and AWS will rely on one of the two being online and Cloudflare will also depend on AWS and the internet will go down and no one will know how to restart it

That's what IRC is for.

(Its Finnish inventor is incidentally working for Google in Stockholm, as per https://en.wikipedia.org/wiki/Jarkko_Oikarinen)

Re: GCP Outage

#478

Post by nijave »

oxymoron wrote: post #67
ransom1538 wrote: post #21

Earlier quoted context omitted.

Why can't companies be honest with being down. It helps us all out so we don't spend an hour internalizing. We are truly in gods hands. $ prod Fetching cluster endpoint and auth data. ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=503, message=Visibility check was unavailable. Please retry the request and contact support if the problem persists

Because a lot of the time, not everyone is impacted, as the systems are designed to contain the "blast radius" of failures using techniques such as cellular architecture and [shuffle sharding]( https://aws.amazon.com/builders-library/workload-isolation-u... ). So sometimes a service is completely down for some customers and fully unaffected for other customers.

It's not rocket science. Put a message up "The service is currently degraded and some users may see errors"

Re: GCP Outage

#479

Post by herpderperator »

When Google said GCP is "down", did it affect entire availability zones within a region? For people who designed redundant infrastructure, did your backup AZs/regions keep your systems online?

Re: GCP Outage

#480

Post by nijave »

deepsun wrote: post #428
hnuser123456 wrote: post #127

Earlier quoted context omitted.

"there is a 5% chance your instance is down" is still a partial outage. A green check should only mean everything (about that service) is working for everyone (in that region) as intended. Downdetector reports started spiking over an hour ago but there still isn't a single status that isn't a green checkmark on the status page.

With highly distributed services there's always something failing, some small percentage.

Sure but you can still put a message up when it's some over some like errors are 50% higher than normal (maybe the SLO is 99.999% of requests are processed successfully)
Post reply on HN