Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Is Down

news.ycombinator.com

Post reply on HN
Print view View original

521–530 of 630 posts

Re: Google Cloud Is Down

#521

Post by scarface74 »

molesy wrote: post #459
jonhohle wrote: post #390

Earlier quoted context omitted.

> it was well-known that all the AWS management was hosted out of a single zone, and there were several days we had to cancel class because us-east-1 had an outage Maybe you mean region, because there is no way that AWS tools were ever hosted out of a single zone (of which there are 4 in us-east-1). In fact, as of a few years ago, the web interface wasn’t even a single tool, so it’s unlikely that there was a global o…

Quoting https://docs.aws.amazon.com/general/latest/gr/rande.html "Some services, such as IAM, do not support Regions; therefore, their endpoints do not include a Region." There was a partial outage maybe a month and a half ago where our typical AWS Console links didn't work but another region did. My understanding is that if that outage were in us-east-1 then making changes to IAM roles wouldn't have worked.

The original poster said that none of AWS services are in a single AZ, the quote you referenced says that IAMs do not support regions.

Your quote cd mean two things.

- that IAM services are hosted in one region (not one AZ)

And/Or

- that IAM is for the entire account not per region like other services (which is true)

Re: Google Cloud Is Down

#522

Post by dingo_bat »

_xerxes_ wrote: post #349

Nest is down too, not surprising given they are part of Google. What I don't understand is why I can't still control my devices over my local network. Why does the system even require access to Google servers?

Because you bought it. In contrast, Philips hue keeps working even if google is down, philips is down, your internet is down, or the entire internet infrastructure has collapsed.

Re: Google Cloud Is Down

#523

Post by jefftk »

nullwasamistake wrote: post #518
agwa wrote: post #506

Earlier quoted context omitted.

The 16 9's are for durability , not availability. AWS is not saying S3 will never go down; they're saying it will rarely lose your data.

This number is still total bullshit. They could lose a few kb and be above that for centuries

None of the big cloud providers have unrecoverably lost hosted data yet, despite sorting vast volumes, so this doesn't seem BS to me.

Re: Google Cloud Is Down

#524

Post by mkl »

aaronblohowiak wrote: post #513
angstrom wrote: post #493

Earlier quoted context omitted.

Netflix actually added the additional AZs because of a prior outage that did take them down. "After a 2012 storm-related power outage at Amazon during which Netflix suffered through three hours of downtime, a Netflix engineer noted that the company had begun to work with Amazon to eliminate “single points of failure that cause region-wide outages.” They understood it was the company’s responsibility to ensure Netflix…

We went multi-region as a result of the 2012 inc. source: I now manage the team responsible for performing regional evacuations (shifting traffic and scaling the savior regions).

That sounds fascinating! How often does your team have to leap into action?

Re: Google Cloud Is Down

#526

Post by manigandham »

obeattie wrote: post #497
manigandham wrote: post #484

Earlier quoted context omitted.

How much cloud spend do you control? That's the reality of how decisions are made.

Many millions of dollars per year. I care about how my providers behave when they have issues, and I can't see why you think it's not at all relevant.

> "why you think it's not at all relevant"

Nobody said this.

> "I care about how my providers behave when they have issues"

We all do.

As the other commenters stated, the communication is poor because the clouds are still growing rapidly and there's not much reason to be better. We might also be underestimating just how much more better service would cost and whether it's worth the revenue loss (if any). Are you really going to shift all of your spend overnight because of an outage? And where are you going to go?

The reality of these decisions is far more nuanced than it may seem and the current state of support is probably already optimized for revenue growth and customer retention.

Re: Google Cloud Is Down

#527

Post by aaronblohowiak »

mkl wrote: post #524
aaronblohowiak wrote: post #513

Earlier quoted context omitted.

We went multi-region as a result of the 2012 inc. source: I now manage the team responsible for performing regional evacuations (shifting traffic and scaling the savior regions).

That sounds fascinating! How often does your team have to leap into action?

We don’t usually discuss the frequency of unplanned failovers, but I will tell you that we do a planned failover at least every two weeks. The team also uses traffic shaping to perform whole system load tests with production traffic, which happens quarterly.

Re: Google Cloud Is Down

#528

Post by rum3 »

ganeshkrishnan wrote: post #480
kerng wrote: post #342

Earlier quoted context omitted.

I'm not sure why you are downvoted - seems like a reasonable insight and explanation for the drop in quality and weird decisions Google is making recently.

I have mentioned this multiple time: Any criticism of Google is met with barrage of downvotes. I guess all the googlers hang around here and they are usually commenting with throwaways.

Google seems to be sort of like a sect of narcissists.

Re: Google Cloud Is Down

#529

Post by [deleted] »

Yrlec wrote: post #322

Now is a good time to point out that the SLA of Google Cloud Storage only covers HTTP 500 errors: https://cloud.google.com/storage/sla . So if the servers are not responding at all then it's not covered by the SLA. I've brought this to their attention and they basically responded that their network is never down.

[deleted]

Re: Google Cloud Is Down

#530

Post by dodobirdlord »

solidasparagus wrote: post #450
dodobirdlord wrote: post #360

Earlier quoted context omitted.

S3 buckets are a global namespace, so control plane operations have to be single-homed. As an example, global consensus has to be reached before returning a success response for bucket creation to ensure that two buckets can't be created with the same name.

But isn't CreateBucket the single s3 operation where you need global consistency?

As far as I know bucket policy operations also require global consistency.
Post reply on HN