Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud networking issues in us-east1

status.cloud.google.com

Post reply on HN
Print view View original

141–150 of 341 posts

Re: Google Cloud networking issues in us-east1

#141

Post by jsty »

lwb wrote: post #131
CaptainJustin wrote: post #101

Earlier quoted context omitted.

It's quite common in cloud solution design to design for failure. One of the common assumptions that we hold to is that one region may go down. Other examples: Assume an instance of an app can go down. Assume a VM can go down. Assume a DC can go down. This is not to excuse the downtime in any way.

Do people ever worry that an entire cloud provider may go down, or is that too unlikely of a case?

However much we technical people might salivate at the prospect of designing a multi-cloud solution, for the vast majority of businesses it simply isn't worth the cost / complexity. I'd wager 90-something percent of applications could suffer multi-hour outages without impacting business function to any measurable degree.

Plus the fact that without serious investment, you're probably more liable to decrease availability by going multi-cloud thanks to the increased system complexity.

Re: Google Cloud networking issues in us-east1

#142

Post by debaserab2 »

lwb wrote: post #131
CaptainJustin wrote: post #101

Earlier quoted context omitted.

It's quite common in cloud solution design to design for failure. One of the common assumptions that we hold to is that one region may go down. Other examples: Assume an instance of an app can go down. Assume a VM can go down. Assume a DC can go down. This is not to excuse the downtime in any way.

Do people ever worry that an entire cloud provider may go down, or is that too unlikely of a case?

I don’t think it’s happened (yet) although some of the earlier outages when AWS was younger were pretty far reaching. I think all of S3 has gone down a time or two.

Re: Google Cloud networking issues in us-east1

#143

Post by harshreality »

Hacker News: The real status page and help desk for the internet.

Do companies realize how absurd this is?

ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.

Re: Google Cloud networking issues in us-east1

#144

Post by geogram »

tlynchpin wrote: post #137
geogram wrote: post #108

Longer than 4 hours. We have stackdriver setup to monitor uptime/latency and its been acting up since 2am PST.

ObPedant: notice in google's status page "...as of Tuesday, 2019-07-02 09:11 US/Pacific." This notation is useful because it's stable year round. I don't recommend 'PDT', instead colloquially 'out here on the left coast' or specifically US/Pacific.

Thanks. Good point. Regardless, gcloud has been having issues for nearly 12hours. (timezone agnostic)

Re: Google Cloud networking issues in us-east1

#145

Post by mbesto »

benbro wrote: post #98
_Codemonkeyism wrote: post #95

Earlier quoted context omitted.

Yes.

Is it practical to use several providers when egress is so expensive?

No. And there's been a lot of talk recently about multi-provider being the right strategy to mitigate downtime, which IMHO is a farce peddled by expensive consultants. The parent comment is correct - this is why availability zones and regions have been established by each provider.

For the large majority of businesses investing in infrastructure-as-code far outweighs any crazy HA, redundant, multi-provider, whizzbang whatever setup you may have.

Re: Google Cloud networking issues in us-east1

#146

Post by deanCommie »

outworlder wrote: post #61
mrmattyboy wrote: post #30

To whomever commented something like 'laughs in AWS' (comment was removed before I submitted the comment)... please don't... glass house and all that... but I also share the same glass house as you.. I don't want bad luck ... and it's only a fluke that this happened to google in eu-east1 and not AWS in X region and then you (and I) would be having a time of hell! :/

Google seems to be more forthcoming with their issues. We have seen incidents in AWS where the status never got updated, but support confirmed issues.

Show me a GCP post-mortem that's as detailed and proactive about future improvement as https://status.aws.amazon.com/s3-20080720.html

Their last one was laughable in it's lack of self-awareness.

Re: Google Cloud networking issues in us-east1

#147

Post by samcday »

alteria wrote: post #126
samcday wrote: post #69

Earlier quoted context omitted.

Which, IMO, is actually a big problem. AFAIK Amazon are running a lot of actual production loads on AWS. Dogfooding can be extremely valuable, especially if a massive portion of your staff have the same profession as your target market. I've been using Google Cloud in a new role I started recently. There's definitely some parts of GCP I like, but whenever I use the Web Console I get the distinct impression nobody at…

I'm curious - what are some examples of the warts you encounter?

Things like:

* filtering traces by services has been broken in App Engine flex environments for more than a year. * copy/pasting identifiers between places is a nightmare * their IAM design is somehow worse than AWS. It’s so impressively bad I can’t even be mad. My favourite part of their IAM approach is how they have consolidated a majority of the IAM controls in the IAM page, but then random services like GCS have it defined elsewhere. * not able to do basic time zooming of metric grafs on App Engine dashboard. * multi-account paper cuts. Almost everyone on my team has their personal and work google accounts logged in. Whenever I send them a link to a dashboard or whatever, they end up getting a permission denied, without fail.

These are all just off the top of my head. Many of them seem silly and minor (and they are!) but there’s enough of them that I kinda dread doing anything in the Cloud console now. I need to take more time to get productive in the gcloud CLI I guess...

Re: Google Cloud networking issues in us-east1

#148

Post by boulos »

username444 wrote: post #58

Cloudflare was returning a 502 this morning, wonder if they're related. Lots and lots of sites down for about an hour, including all of Shopify.

As jgrahamc (Cloudflare CTO) noted below, these aren't related. They had a push that they rolled back, we lost some fiber links.

Re: Google Cloud networking issues in us-east1

#149

Post by notatoad »

harshreality wrote: post #143

Hacker News: The real status page and help desk for the internet. Do companies realize how absurd this is? ETA: It seems someone at Google had a change of heart, and most of what boulos posted in this thread has been added as updates to the official google status page. Better late than never, I guess, especially if this is the start of a trend in outage reporting.

seriously, they've got a text field on the official status page, why not put the text boulos posted here in that instead of the meaningless text they've got there?
Post reply on HN