Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

141–150 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#141

Post by marcinzm »

joshuamorton wrote: post #128
michaelt wrote: post #120

Earlier quoted context omitted.

At a smaller, less corporate company the engineer/public comms person dividing line would not be so ossified the divide couldn't be bridged when the situation called for it.

I think the broader point here is that they (whomever they is) don't think the situation calls for it. That's why they said the next update would be on Monday on Friday. You may feel that's a bad decision, but I doubt that people are in a panic because they can't push out an update that would not be noticably different from the last one. Just to clarify, what should this update contain?

>Just to clarify, what should this update contain?

"We're working on it", possibly an ETA for a fix or some details. Technically it's fluff but people are not machines and the update is for people. We like the feeling that people are working on a fix, that people care and that the end is in sight. It makes the situation less stressful and, as for why Google should care, less stressed engineers won't bad mouth Google as much after the fact.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#142

Post by chongli »

chrisseaton wrote: post #137
justinsb wrote: post #50

Earlier quoted context omitted.

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy ( https://cloud.google.com/terms/ ), whose intent is to give you some assurances. (I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

But hey, it's all spelled out in the policy, so don't say we didn't warn you!

Caveat emptor, folks.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#143

Post by davemp »

antpls wrote: post #91
justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the…

I’ll suggest considering whether entities enamored with centralizing ideals are more likely to fail to properly realize the robustness of a distributed system.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#144

Post by 8ytecoder »

chrisseaton wrote: post #137
justinsb wrote: post #50

Earlier quoted context omitted.

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy ( https://cloud.google.com/terms/ ), whose intent is to give you some assurances. (I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

>Subject to Section 7.2

Which is the deprecation policy. (I mean I share your frustration with Google's what-appears-to-be-at-least haphazard policy of shutting down services instead of trying to gain traction. But, let's not misrepresent what they say).

Re: Google Kubernetes Engine's third consecutive day of service disruption

#145

Post by asdfasgasdgasdg »

hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

Not to minimize here (well, yes, a little), but this was a UI-only outage, from what I can tell. You could still create the pools from the command-line. It doesn't seem unreasonable to have a single, global UI server, as long as the API gateway is distributed and not subject to global outages.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#146

Post by joshuamorton »

chrisseaton wrote: post #137
justinsb wrote: post #50

Earlier quoted context omitted.

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy ( https://cloud.google.com/terms/ ), whose intent is to give you some assurances. (I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

The preceding line is the imprtant part. Which is essentially

"Subject to the deprecation policy [which says that Google will give at least 1 year notice before cancelling services], Google may discontinue..."

In other words, at any time, google can give you a years notice.

(I work at Google, but am not a lawyer and this isn't official in any capacity).

Please don't selectively quote things out of context to give a misleading impression.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#147

Post by uluyol »

chrisseaton wrote: post #137
justinsb wrote: post #50

Earlier quoted context omitted.

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy ( https://cloud.google.com/terms/ ), whose intent is to give you some assurances. (I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

Nice job cherry picking text.

> 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue > any Services or any portion or feature for any reason at any time without > liability to Customer.

Let's take a look at Section 7.2:

> 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > make backwards incompatible changes to the Services specified at the URL in > the next sentence. Google will use commercially reasonable efforts to continue > to operate those Services versions and features identified at > https://cloud.google.com/terms/deprecation without these changes for at least > one year after that announcement, unless (as Google determines in its > reasonable good faith judgment): > > (i) required by law or third party relationship (including if there is a change > in applicable law or relationship), or > > (ii) doing so could create a security risk or substantial economic or material > technical burden. > > The above policy is the "Deprecation Policy."

To me that looks like a reasonable deprecation policy.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#148

Post by chrisseaton »

8ytecoder wrote: post #144
chrisseaton wrote: post #137

Earlier quoted context omitted.

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

>Subject to Section 7.2 Which is the deprecation policy. (I mean I share your frustration with Google's what-appears-to-be-at-least haphazard policy of shutting down services instead of trying to gain traction. But, let's not misrepresent what they say).

I thought that 'subject to 7.2' meant that they can use the escape-hatch there 'substantial economic or material technical burden'? They can list anything under that.

I don't think it's wrong - they can deprecate any service they want to do whenever they want, unless people have paid for and signed a contract that says otherwise which I guess people aren't doing.

But the policy doesn't really guarantee anything at all does it, due to the reference escape-hatches? It might as well not exist?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#150

Post by shareometry »

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly:

1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN).

2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.

3) The sum total of information about this incident can be found as a few one or two sentence blurbs on Google's blog. No explanation nor outline of scope for affected regions and services has been provided.

4) Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems.

5) Some users here are reporting that they have received no response from GCP support, even over a time span of 40+ hours since the support request was submitted.

6) Google says they'll provide some information when the next business day rolls around, roughly 4 days after the start of the problem.

I really do want to make sure I'm understanding this situation. Please do correct me if I got something wrong in this summary.

Post reply on HN