Google Kubernetes Engine's third consecutive day of service disruption

#1

Post by rlancer » Sun, Nov 11, 2018, 8:47 PM UTC

Google Kubernetes Engine's third consecutive day of service disruption
status.cloud.google.com

Re: Google Kubernetes Engine's third consecutive day of service disruption

#2

Post by rlancer » Sun, Nov 11, 2018, 8:53 PM UTC

Status page is inaccurate as issues doesn't only affect the web UI, the same operations are not functioning via the CLI.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#3

Post by [deleted] » Sun, Nov 11, 2018, 9:16 PM UTC

[deleted]

Re: Google Kubernetes Engine's third consecutive day of service disruption

#4

Post by base698 » Sun, Nov 11, 2018, 9:17 PM UTC

rlancer wrote: post #2

Status page is inaccurate as issues doesn't only affect the web UI, the same operations are not functioning via the CLI.

What operations? Status just shows node pool creation.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#5

Post by pm90 » Sun, Nov 11, 2018, 9:18 PM UTC

rlancer wrote: post #2

Status page is inaccurate as issues doesn't only affect the web UI, the same operations are not functioning via the CLI.

Its kinda strange that HN seems to be the most effective way to give feedback to Google Cloud :/

Re: Google Kubernetes Engine's third consecutive day of service disruption

#6

Post by regnerba » Sun, Nov 11, 2018, 9:24 PM UTC

Is this just about creating new pools? I haven't noticed an issue with our existing pools scaling.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#7

Post by rlancer » Sun, Nov 11, 2018, 9:25 PM UTC

base698 wrote: post #4

rlancer wrote: post #2

Status page is inaccurate as issues doesn't only affect the web UI, the same operations are not functioning via the CLI.

What operations? Status just shows node pool creation.

Can not create a new Clusters or Node Pool and can not resize exiting Node Pools, as far as users are reporting it's happening in all regions too.

Error message when creating a new Cluster:

Deploy error: Not all instances running in IGM after 35m7.509000994s. Expect 1. Current errors: [ZONE_RESOURCE_POOL_EXHAUSTED]: Instance 'gke-cluster-3-pool-1-41b0abf8-73d7' creation failed: The zone 'projects/url-shortner-218503/zones/us-west2-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later. - ; .

Re: Google Kubernetes Engine's third consecutive day of service disruption

#8

Post by closeparen » Sun, Nov 11, 2018, 9:27 PM UTC

Doesn’t GKE “just” run an independent Kubernetes cluster on customer VMs? How is a widespread outage like this possible?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#9

Post by rlancer » Sun, Nov 11, 2018, 9:31 PM UTC

closeparen wrote: post #8

Doesn’t GKE “just” run an independent Kubernetes cluster on customer VMs? How is a widespread outage like this possible?

GKE gives you a fully managed Master Node.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#10

Post by regnerba » Sun, Nov 11, 2018, 9:33 PM UTC

closeparen wrote: post #8

Doesn’t GKE “just” run an independent Kubernetes cluster on customer VMs? How is a widespread outage like this possible?

GKE does the creation of the VMs and setup of them, joining them to the cluster and applying labels for example.

The specific issue appears to be about creating new "node pools". Creating standard VMs in GCP works fine however, so this is specific to GKE and their internal tooling that integrates with the rest of GCP.

GKE doesn't (at least to my knowledge) allow you to create VMs separately and join them to the cluster in any kind of easy fashion.