Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

101–110 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#101

Post by ramphastidae »

izacus wrote: post #57
marcinzm wrote: post #26

Earlier quoted context omitted.

Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.

It's weekend, why wouldn't you take it off? It's just silly software.

A lot of people's businesses, reputations and livelihoods depend on "silly software," not to mention that they are paying customers themselves.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#102

Post by hartem_ »

hacknat wrote: post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

I’d be curious to know what alternatives are you considering at this point?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#103

Post by scarface74 »

devonkim wrote: post #80
mey wrote: post #44

Earlier quoted context omitted.

First, determine your tolerance. How much does downtime per min cost you? How long can you be down? A lot of the time it may be cheaper to apologize to your customers than build a truly reliable system. Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage? There are formal academic w…

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough. Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “pri…

That does bring up an interesting point. In hindsight, we already have duplicate infrastructure - a dev account and a production account. Why in the world was it decided to put both accounts in the same region?

The separate account was setup partially on my insistence but it was set up in the same region.

If needed, we could have done VPC peerings across regions. (https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...)

Re: Google Kubernetes Engine's third consecutive day of service disruption

#104

Post by antpls »

carbocation wrote: post #99
antpls wrote: post #91

Earlier quoted context omitted.

The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the…

> I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA? I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).

Tried on us-east, us-north, europe-west, also tried asia, with different instance sizes and with both UI and CLI. None worked for me.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#106

Post by scarface74 »

user5994461 wrote: post #100
devonkim wrote: post #80

Earlier quoted context omitted.

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough. Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “pri…

The baseline is that it takes 12 dedicated people across the world to run a 24/7 support operation. Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.

Most small companies on AWS with revenue outsource support to an MSP.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#107

Post by tehlike »

blazespin wrote: post #96
noselasd wrote: post #65

Earlier quoted context omitted.

The issue people have with it is that it's global, not regional, indicating that there are dependencies in the entire architecture that people does not expect to be there.

Yes, hello, canaries anyone

Plenty of bugs happen despite canaries.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#108

Post by Waterluvian »

marcinzm wrote: post #78
NicoJuicy wrote: post #77

Earlier quoted context omitted.

The default is : https://status.cloud.google.com/incident/container-engine/18... People who respond here could be employees of Google, caring about it and respond here because they know it. What he can mention ( a lot of people are working on it) is what you can suspect when something is going down. All other cloud providers do the same.

The default you linked to has not been updated in 2 days... which is my whole point regarding having to rely on hacker news for any status updates. edit: The default is also only about the UI issue and there's no issue tracker for the broader non-UI disruptions going on since Friday.

Even an update of "no change" is tremendously valuable.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#109

Post by sascha_sl »

mcqueenjordan wrote: post #97
jorams wrote: post #86

Earlier quoted context omitted.

I'm not sure where you get this idea, but AWS has definitely had global outages. 1.5 years ago there were massive global issues with S3, causing even their own status dashboard to malfunction. edit: I stand corrected. Apparently the S3 outage wasn't global, though its effects were. Meanwhile, this outage has only really been noticeable to ops teams, since it doesn't affect existing nodes or anything outside GKE. It's…

On February 28th, 2017, S3 had issues in us-east-1, not globally. It just happens that most customers create their buckets in us-east-1. (It’s the default bucket creation region.)

Correct, but quite a few AWS services are built on S3, and some apparently weren't using buckets per region.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#110

Post by rlancer »

rlancer wrote: post #62
justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

Creating clusters via the UI is still not working for me.

UPDATE: Created a Cluster successfully in Australia... Still not able to do so in the US.
Post reply on HN