Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

151–160 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#151

Post by tomcam »

justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

Thanks for jumping in here on your own time. The following question is not meant to be hostile, it is merely curiosity. Isn’t this supposed to be the kind of thing that monitoring and diagnostics software should find automatically? Serious question, not meant to embarrass you.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#152

Post by chrisseaton »

joshuamorton wrote: post #146
chrisseaton wrote: post #137

Earlier quoted context omitted.

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

The preceding line is the imprtant part. Which is essentially "Subject to the deprecation policy [which says that Google will give at least 1 year notice before cancelling services], Google may discontinue..." In other words, at any time, google can give you a years notice. (I work at Google, but am not a lawyer and this isn't official in any capacity). Please don't selectively quote things out of context to give a m…

But what do these things mean?

> commercially reasonable

> substantial economic or material technical burden

Is one engineer working on an old service to keep it alive commercially reasonable or a substantial burden? I don't know. Do you?

In practice this policy lets them shut off anything they want any time they want. Again it's their playground they can do what they want unless they signed a contract saying they'd do something else for you so I don't have a problem with it.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#153

Post by jorblumesea »

stevehawk wrote: post #140
antpls wrote: post #91

Earlier quoted context omitted.

The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the…

This is unfortunately the norm. Like when AWS S3 went down (but couldn't update its own status images because they're in S3 and we all laughed) and along with it went Alexa, lambda, and every other service dependent on S3.

S3 is really the one of the few services on aws that can do that unfortunately. It has no concept of zone/region, it's truly global. To me it seems like a serious design flaw, as everything else in aws is striped by region, but not sure why exactly it was built like that.

edit:

nvm s3 has regions, it's the bucket names that are global.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#154

Post by tomcam »

hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

I believe this is a fair question. I’d really like to understand what Google thinks about this.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#155

Post by Jedi72 »

uluyol wrote: post #147
chrisseaton wrote: post #137

Earlier quoted context omitted.

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

Nice job cherry picking text. > 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue > any Services or any portion or feature for any reason at any time without > liability to Customer. Let's take a look at Section 7.2: > 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > make backwards incompatible changes to the Services specified at the URL in > the next senten…

It's ok I guess but still lets them turn it off if, in their judgement, its an economic burden i.e. costing them money.

If they ever do deprecate something people have built on though they're gonna get absolutely crucified. That's probably better protection than any terms of service.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#156

Post by Marsymars »

user5994461 wrote: post #100
devonkim wrote: post #80

Earlier quoted context omitted.

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough. Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “pri…

The baseline is that it takes 12 dedicated people across the world to run a 24/7 support operation. Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.

What's the math/logic to get to 12 people?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#157

Post by scoot »

uluyol wrote: post #147
chrisseaton wrote: post #137

Earlier quoted context omitted.

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

Nice job cherry picking text. > 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue > any Services or any portion or feature for any reason at any time without > liability to Customer. Let's take a look at Section 7.2: > 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > make backwards incompatible changes to the Services specified at the URL in > the next senten…

> To me that looks like a reasonable deprecation policy.

It might be, until they jack up the prices 15X with limited notice (looking at you, Google maps [1]). No deprecation needed, just force users off the platform unless they're willing to pay a massive premium.

[1] https://www.google.com/search?q=google+maps+price+increase

Re: Google Kubernetes Engine's third consecutive day of service disruption

#158

Post by rlancer »

asdfasgasdgasdg wrote: post #145
hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

Not to minimize here (well, yes, a little), but this was a UI-only outage, from what I can tell. You could still create the pools from the command-line. It doesn't seem unreasonable to have a single, global UI server, as long as the API gateway is distributed and not subject to global outages.

Was certainly not UI only

Re: Google Kubernetes Engine's third consecutive day of service disruption

#159

Post by kuyan »

Jedi72 wrote: post #155
uluyol wrote: post #147

Earlier quoted context omitted.

Nice job cherry picking text. > 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue > any Services or any portion or feature for any reason at any time without > liability to Customer. Let's take a look at Section 7.2: > 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > make backwards incompatible changes to the Services specified at the URL in > the next senten…

It's ok I guess but still lets them turn it off if, in their judgement, its an economic burden i.e. costing them money. If they ever do deprecate something people have built on though they're gonna get absolutely crucified. That's probably better protection than any terms of service.

> It's ok I guess but still lets them turn it off if, in their judgement, its an economic burden i.e. costing them money.

If a service Google runs is losing money, what reason would they have to not shut it down?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#160

Post by joshuamorton »

chrisseaton wrote: post #152
joshuamorton wrote: post #146

Earlier quoted context omitted.

The preceding line is the imprtant part. Which is essentially "Subject to the deprecation policy [which says that Google will give at least 1 year notice before cancelling services], Google may discontinue..." In other words, at any time, google can give you a years notice. (I work at Google, but am not a lawyer and this isn't official in any capacity). Please don't selectively quote things out of context to give a m…

But what do these things mean? > commercially reasonable > substantial economic or material technical burden Is one engineer working on an old service to keep it alive commercially reasonable or a substantial burden? I don't know. Do you? In practice this policy lets them shut off anything they want any time they want. Again it's their playground they can do what they want unless they signed a contract saying they'd…

I think you're ascribing an unreasonable amount of bad faith here, and, to rephrase what I had here before, you're approaching this from an engineering perspective, not a legal one. And that's not how those things work.

To be clear, that policy is a contract. And those things would be decided by a jury. And if my understanding is correct, the reasonable person standard applies. So you can answer this yourself, do you think a reasonable person would believe that your interpretation is valid?

If not, why mention it?

Post reply on HN