Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

211–220 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#211

Post by matwood »

scoot wrote: post #157
uluyol wrote: post #147

Earlier quoted context omitted.

Nice job cherry picking text. > 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue > any Services or any portion or feature for any reason at any time without > liability to Customer. Let's take a look at Section 7.2: > 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > make backwards incompatible changes to the Services specified at the URL in > the next senten…

> To me that looks like a reasonable deprecation policy. It might be, until they jack up the prices 15X with limited notice (looking at you, Google maps [1]). No deprecation needed, just force users off the platform unless they're willing to pay a massive premium. [1] https://www.google.com/search?q=google+maps+price+increase

The maps price gouge is yet another reason I will not use google services for anything but ancillary services.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#212

Post by icelancer »

dilyevsky wrote: post #195
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

We had an issue a few weeks back where all nodes in west1-a could not pull docker images. Google support was pinballing P1 issue around the globe and across multiple teams for a few days untill I root caused it for them - turned out to be gce service account issues affecting entire zone. 2 days to rollback (no status page update). I know nobody gives a fuck but can’t help but feel vindicated as an ex google sre.

I think a lot of people give a fuck here; I do, at least. Thanks for outlining it, these things are fascinating (to me anyway, who has never worked in IT/ops).

Re: Google Kubernetes Engine's third consecutive day of service disruption

#213

Post by joshuamorton »

doodliego wrote: post #199
joshuamorton wrote: post #128

Earlier quoted context omitted.

I think the broader point here is that they (whomever they is) don't think the situation calls for it. That's why they said the next update would be on Monday on Friday. You may feel that's a bad decision, but I doubt that people are in a panic because they can't push out an update that would not be noticably different from the last one. Just to clarify, what should this update contain?

Compare Amazon's responses during their S3 outage: https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ (toward the bottom) What's the better communication plan: detailed, hourly updates or terse, one-line blog posts scattered across several days?

I think this comes down to the two outages being not at all similar. An s3 outage affects practically every Amazon customer. This affects a relatively small number of go customers.

And I'm confused about what was good about that response. That article is about how the s3 outage caused so many issues that Amazon couldn't update their status dashboard to inform users at all.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#214

Post by jkaplowitz »

lovich wrote: post #182
kuyan wrote: post #159

Earlier quoted context omitted.

> It's ok I guess but still lets them turn it off if, in their judgement, its an economic burden i.e. costing them money. If a service Google runs is losing money, what reason would they have to not shut it down?

With this terms of service, none. Which is why people don't trust them. If I pay you for a service that would take time to migrate off of, and you are making money off me now, I am going to be ripshit if you decide to just turn it off because it's suddenly not making money for you in the short term. Google's done this a lot, and the fact that don't provide concrete time lines in their contract gives even less reason…

It's not about the contract. AWS doesn't even have a deprecation policy in the contract - seriously, GCP provides more legally binding guarantees than AWS. It's about trust.

People look at AWS's track record, and trust that. People look at Google's track record, overlook what to an inside-the-company Googler perspective are dramatically significant organizational boundaries or product lifecycle definitions that are very poorly communicated outside the company, mentally apply reputational damage from one part of Google (or from a preview-stage GCP product) to a different part of the company (or to a generally available GCP product), and don't trust that.

Google has always been worse at externally facing PR than at the internal reality, even when I worked there (2011-2015). Major company weakness.

But the internal reality inside GCP, perceptions aside, is pretty good even now.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#215

Post by locusm »

Do not use GCP without paying for support. We have had resource allocation errors for weeks, as have a lot of other people. Check out the posts in their forum where folk on basic support get zero love. https://groups.google.com/forum/?utm_medium=email&utm_source...

Re: Google Kubernetes Engine's third consecutive day of service disruption

#216

Post by ToFab123 »

navinsylvester wrote: post #193
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform. So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had ra…

I don't understand why someone would choose to deploy anything mission critical without having an support contract with the ISP, the manufacturer of the the software etc.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#217

Post by [deleted] »

jkaplowitz wrote: post #206
Jedi72 wrote: post #121

Earlier quoted context omitted.

They work for Google so obviously they are much smarter than you. If theres a problem its probably the customers fault. /sarcasm

Many of the tier 1 GCP support reps work for external vendors nowadays, which is probably part of the problem. During my time on the GCE team (note I don't work at Google now) I knew multiple full-time Google employee support reps, including some still at the company. They have the good attitude and deep knowledge you'd hope for. The problem is simply about how Google scales their GCP support org. To be completely cl…

[deleted]

Re: Google Kubernetes Engine's third consecutive day of service disruption

#218

Post by scarface74 »

Say I were a CTO (I’m nowhere near it), why would I choose GCP over AWS or Azure? Even if after doing a technical assessment and I thought that GCP was technically slightly better, if something happened, the first question I would be asked is “why did you choose GCP over AWS?”

No one would ever ask why you chose AWS. The old “no one ever got fired for buying IBM”.

Even if you chose Azure because you’re a Microsoft shop, no one would question your choice of MS. Besides, MS is known for thier enterprise support.

From a developer/architect standpoint, I’ve been focused the last year on learning everything I could about AWS and chose a company that fully embraced it. AWS experience is much more marketable than GCP. It’s more popular than Azure too, but there are plenty of MS shops around that are using Azure.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#219

Post by marcinzm »

ToFab123 wrote: post #216
navinsylvester wrote: post #193

Earlier quoted context omitted.

We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform. So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had ra…

I don't understand why someone would choose to deploy anything mission critical without having an support contract with the ISP, the manufacturer of the the software etc.

Simple, the cost of an outage is less than the cost of a support contract. Very few things are really mission critical as in they can never go down. Rather they simply have a cost to going down and you can choose to pay that one way or another.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#220

Post by tejohnso »

shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

> For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement)

What blog statement are you referring to? I don't see any such statement. Can you provide a link?

The OP incident status issue says "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI". It also says "Affected customers can use gcloud command in order to create new Node Pools."

So it sounds like a web interface problem, not a severely limiting, backend systems problem with global scope.

Also, the report says "The issue with Google Kubernetes Engine Node Pool creation through the Cloud Console UI had been resolved as of Friday, 2018-11-09 14:30 US/Pacific". So the whole issue lasted about 10 hours, not three whole days.

> Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems

I don't see much of that.

Post reply on HN