Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

241–250 of 341 posts

Re: Google Cloud networking issues in us-east1

#241

Post by dodobirdlord »

jsjohnst wrote: post #229
lwb wrote: post #131

Earlier quoted context omitted.

Do people ever worry that an entire cloud provider may go down, or is that too unlikely of a case?

In the past ten years: It’s happened more than once with Azure and GCP. I think it happened once with AWS, but not positive there.

AWS had a multi-hour total S3 outage in us-east-1 in February 2017 that knocked out a huge number of things mostly because it turns out that a huge share of their customers run in only 1 region and it's us-east-1. Things mostly continued to work in other regions.

I recall Azure had some sort of multi-region database failover disaster that took several regions offline, and GCP has had several global elevated latency/error rate events, but I don't think that any cloud provider has been "down" in the sense that the word is usually used.

Re: Google Cloud networking issues in us-east1

#242

Post by jhgg »

boulos wrote: post #148
username444 wrote: post #58

Cloudflare was returning a 502 this morning, wonder if they're related. Lots and lots of sites down for about an hour, including all of Shopify.

As jgrahamc (Cloudflare CTO) noted below, these aren't related. They had a push that they rolled back, we lost some fiber links.

Cloudflare took us down this morning, but also shielded us from the impact of this fiber cut, due to direct peering with google (I’m assuming over different fiber paths.)

Re: Google Cloud networking issues in us-east1

#243

Post by mehrdadn »

niyazpk wrote: post #228
mehrdadn wrote: post #201

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

As more businesses move their compute to the cloud, one might predict that more people will be impacted by outages in the large cloud providers. This in turn means that the affected people will start up-voting these threads. Expect these to be more common.

I don't see how this is something that's specific to the last few months though.

Re: Google Cloud networking issues in us-east1

#244

Post by dodobirdlord »

stingraycharles wrote: post #112
opportune wrote: post #94

Earlier quoted context omitted.

Not him but oftentimes cloud outages can be due to issues with the network connections to the datacenter, or power outages. Datacenters also sometimes have other single points of failure such as DNS, but those are within the company's control. https://www.networkworld.com/article/3373646/network-problem... https://www.datacenterknowledge.com/uptime/equinix-power-out...

But data centers are typically designed with network and power failures in mind, not? Isn’t this why these kind of ring based network topologies exist, so that whenever a single network connection fails, it can still easily be routed around?

On a smaller scale, to link up a few datacenters that are a few miles apart? Sure. On a grand scale though, no. Nobody's running an extra undersea cable from Japan to Singapore so that they can have a ring topology. Or trenching a second PBps of cables across the Appalachian Mountains. When something like that gets busted you go and reroute your least important traffic and send out the repair crew.

Re: Google Cloud networking issues in us-east1

#245

Post by dodobirdlord »

gizmo385 wrote: post #90
tscanausa wrote: post #87

Earlier quoted context omitted.

Terrance here from Google Cloud Support. There are only 3 things I can say about this situation. 1) These issues are currently unrelated. 2) We learn a lot from these situations. 3) A lot of these types of issues can be mitigated by running in more then 1 region. I really cant promise that today's situations will never happen again. There are a lot of moving pieces in our system and sometimes there are things outside…

> There are a lot of moving pieces in our system and sometimes there are things outside of Google's control. Are you implying that the cause of this outage is not Google's fault? If so, can you go into more details about that?

> The disruptions with Google Cloud Networking and Load Balancing have been root caused to physical damage to multiple concurrent fiber bundles serving network paths in us-east1, and we expect a full resolution within the next 24 hours.

From the dashboard. Looks like this can be blamed on an Act of Backhoe.

Re: Google Cloud networking issues in us-east1

#247

Post by [deleted] »

boulos wrote: post #136

Disclosure: I work on Google Cloud (but I'm not in SRE, oncall, etc.). As the updates to [1] say, we're working to resolve a networking issue. The Region isn't (and wasn't) "down", but obviously network latency spiking up for external connectivity is bad. We are currently experiencing an issue with a subset of the fiber paths that supply the region. We're working on getting that restored. In the meantime, we've remov…

[deleted]

Re: Google Cloud networking issues in us-east1

#248

Post by IX-103 »

kyrra wrote: post #197
ricardobeat wrote: post #193

Earlier quoted context omitted.

Tangential question: does Google allow employees, not directly tasked with it, to represent the company online as they wish? Most companies I know of have a strict ‘do not speak for the company’ policy.

It's a fine line. We are not allowed to represent Google in any kind of public discussion. But we can talk about some things we do, as long as we state it's our own opinion and we don't represent Google's views.

And don't disclose material nonpublic information (since that would run afoul of insider trading laws).

It's probably okay to say that we know the problem and here are the steps we're taking to mitigate it. It would not be okay to say something with large scale stock price implications for Google it another publicly traded corporation. For instance a Google employee shouldn't say something like "faulty solar panels fried Google's 10 largest data centers and twelve others have been lost to rebel drone strikes", even if false, since it could have a drastic impact on the earnings and future value of Google, Google's customers, and Google's competitors.

Even less obvious things like Google's plans for adding privacy features to the Chromium open source project can have a serious impact (see https://www.barrons.com/articles/google-chrome-privacy-quest...).

Re: Google Cloud networking issues in us-east1

#249

Post by MisterPea »

hunter2_ wrote: post #223
digitalsanctum wrote: post #217

I routinely see notices of outages like this posted on HN while HN itself never seems to be impacted. This begs the question: Where and how is HN hosted in a way that avoids being impacted by widespread network and provider outages?

$ host news.ycombinator.com news.ycombinator.com has address 209.216.230.240 https://whois.arin.net/rest/net/NET-209-216-230-0-1/pft?s=20... M5 Computer Security https://www.m5hosting.com Unrelated: https://begthequestion.info/

The begs the question site is one of my pet peeves. Language is not moderated by a select few who want to claim it, this isn't France.

This is why Ebonics is still a valid form of English - as long as it is used consistently.

If everyone uses "begs the question" and everyone else understands it as "raises the question" then it is perfectly valid.

Post reply on HN