Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

31–40 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#31

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Assuming you have architected multi-region I'm not sure how realistic that scenario is. AWS regions are mostly standalone, I have seen services go down in a region but never globally.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#32

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

"Our company is completely dependent on AWS."

Well not a lot. On the plus side its pretty reliable.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#33

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Roll the dice? What are the consequences for you, especially if you can shift the blame? What are the odds of you having better uptime rolling your own tooling? Can you afford the complexity of multi cloud? Is the added complication worth it?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#34

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Basically, you have three options. You can go full milti cloud, with all the expenses and overhead that entails. You can have everything run in one place but have a plan to switch to a backup system. Or you can look at the overhead and cost associated with those, and decide that it's not worth it. If the business can handle the costs and risks, then any of them can be a valid option.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#35
post #30

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Infrastructure as code. Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services. This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

I think you are downplaying minimal

Re: Google Kubernetes Engine's third consecutive day of service disruption

#36

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

I appreciate all the effort you're putting in and I understand such situations can be stressful but user's having to depend on someone responding on hacker news for status updates seems really amateur for an organization the size of google.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#37

Oh man must be a tough time to be an SRE at google cloud. But... they’re Google. They have been doing internal cloud for years and years. Borg — which K8s is a reimplementation if — has been the heart of Google for so long now you’d think they’d be able to architect their systems to have no outages whatsoever. I mean nobody is perfect but this looks bad.

Goes to show outsourcing infrastructure is more about blame shifting so that when things go wrong its "not our fault" than reducing actual downtime.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#38

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Very little, unless that small business has a big enough budget to design something that spans multiple clouds.

Ultimately it’s a risk/return decision.

“Is going exclusively with AWS/azure/GCP etc a better decision in reliability, financial and mantainability terms than complicating the design to improve resiliency? And will this more complex solution actually improve reliability?”

Re: Google Kubernetes Engine's third consecutive day of service disruption

#39

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

This might be an unpopular opinion but,

Going multi region on AWS should be safe enough.

If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.

You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not feel too bad about my small business’ downtime.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#40

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Using Kubernetes is a good start. It should be easy to migrate your server between EKS and GKE. However data is trickier to move around. So you won’t be immune from all global outages.
Post reply on HN