"The data says engagement is down 46%, I think its time we drop the product."
- Someone at Google right now, probably.
Google Kubernetes Engine's third consecutive day of service disruption
21–30 of 419 posts
Re: Google Kubernetes Engine's third consecutive day of service disruption
#22>Nov 09, 2018 05:59
>We will provide more information by Monday, 2018-11-12 11:00 US/Pacific.
Wait, did the people tasked with fixing this just take the weekend off?
Re: Google Kubernetes Engine's third consecutive day of service disruption
#23A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally.
What would a small business do as a contingency plan?
Re: Google Kubernetes Engine's third consecutive day of service disruption
#24Seems to be some weird underlying issue going on at GCP at the moment. Had cloud build webhooks returning a 500 error. Noticed we were at 255 images and deleting some fixed the issue. Created a P2 ticket about the issue before we managed to solve it and haven't had a response in 40+ hours.
The timeline of this disruption matches when we started experiencing cloud build errors.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#25>Nov 09, 2018 05:59 >We will provide more information by Monday, 2018-11-12 11:00 US/Pacific. Wait, did the people tasked with fixing this just take the weekend off?
The people tasked with fixing this aren't the ones providing the updates.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#26>Nov 09, 2018 05:59 >We will provide more information by Monday, 2018-11-12 11:00 US/Pacific. Wait, did the people tasked with fixing this just take the weekend off?
The people tasked with fixing this aren't the ones providing the updates.
Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#27Been trying to spin up vm instances all day, had to try every single zone just to get one up. Not only is this incredibly harmful to a technology business dependent on this infra, it wasnt obvious to me what the issue was until I tried creating instances. Nothing says, hey resources are constrained here, try this one. Just about ready to bite the bullet and move to aws.
Same here. We have spent 2 days trying to create instances and migrate images just to figure out later they can't start. Right when I convinced our project to get migrated from AWS...
Why on earth would you do that, unless you had huge problems on aws?
Re: Google Kubernetes Engine's third consecutive day of service disruption
#28Oh man must be a tough time to be an SRE at google cloud. But... they’re Google. They have been doing internal cloud for years and years. Borg — which K8s is a reimplementation if — has been the heart of Google for so long now you’d think they’d be able to architect their systems to have no outages whatsoever. I mean nobody is perfect but this looks bad.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#29Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now!
It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.
Re: Google Kubernetes Engine's third consecutive day of service disruption
#30A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?
Infrastructure as code.
Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services.
This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work