Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

81–90 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#81

Post by kazen44 »

shiftnight wrote: post #74

I have a question. At what point does k8s make sense? I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well. Monolith for the win! Opinions?

i agree with you on major points.

Also, migrating to microservices for existing services might not be worth it, especially if you don't operate at a massive scale.

Keep it simple stupid is still a solid design decision, despite all the microservice/container hype.

Most bussinesses only need a couple of servers that provide the service, spread redundantly with a HA capability.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#82

Post by scarface74 »

geggam wrote: post #30
scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

Infrastructure as code. Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services. This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

That’s not how Terraform works. Each provisioner has separate syntax depending on the cloud provider. The template for AWS wouldn’t work on GCP or Azure.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#83

Post by shiftnight »

hacknat wrote: post #76
shiftnight wrote: post #74

I have a question. At what point does k8s make sense? I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well. Monolith for the win! Opinions?

This outage really doesn’t have much to do with K8s.

Maybe so, but you won't be affected by this outage if you never decided to deploy k8s in the first place.

Even if you deploy k8s privately, or over at Amazon, I think there's enough horror stories to make you think twice about the technology.

Then, if it isn't going to be k8s for microservices, what's a more reliable alternative?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#84

Post by marcinzm »

tweenagedream wrote: post #75
marcinzm wrote: post #26

Earlier quoted context omitted.

Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.

This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to? It's likely the fix is checked in and will start roll out on Monday. Disclaimer: I work on Google Cloud and while I believe we could use more words here, this doesn't seem like a huge problem. It's embarrassing the the issue with the ui was shipped, and I'm sure this…

>This is a problem in the web ui creating additional node pools, there is a very simple workaround to use gcloud. What major impact are you referring to?

Based on comments in this thread even gcloud is failing and so are other non-kubernetes services. Which may be inaccurate but there's a lot of people saying the same thing so maybe it is.

You're right however that the linked issue is only about the UI. So Google isn't even tracking the service distribution issue in it's issue tracker much less updating people on. I personally think that's even worse...

Re: Google Kubernetes Engine's third consecutive day of service disruption

#85

Post by trhway »

marcinzm wrote: post #36
justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

I appreciate all the effort you're putting in and I understand such situations can be stressful but user's having to depend on someone responding on hacker news for status updates seems really amateur for an organization the size of google.

>really amateur for an organization the size of google.

There is a reason while Google have been having hard time making inroads in the enterprise cloud. Kind of impedance mismatch between enterprise and the Google style. That 2 stories like high "We heart API" sign on the Google Enterprise building facing 237 just screams about it :)

Re: Google Kubernetes Engine's third consecutive day of service disruption

#86

Post by jorams »

hacknat wrote: post #45
eknkc wrote: post #39

Earlier quoted context omitted.

This might be an unpopular opinion but, Going multi region on AWS should be safe enough. If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days. You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not fee…

Yeah, AWS has never had a global outage, Google has had 2 now.

I'm not sure where you get this idea, but AWS has definitely had global outages. 1.5 years ago there were massive global issues with S3, causing even their own status dashboard to malfunction.

edit: I stand corrected. Apparently the S3 outage wasn't global, though its effects were.

Meanwhile, this outage has only really been noticeable to ops teams, since it doesn't affect existing nodes or anything outside GKE. It's definitely concerning and the fix is taking far too long, but as far as global outages go the impact is relatively minor.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#87

Post by kenhwang »

yeukhon wrote: post #73
tehlike wrote: post #59

Earlier quoted context omitted.

2 outage in 5 years sounds pretty low, to be honest. Disclaimer: google employee in ads, who worked on many many fires throughout the years, but talking from my personal perspective and not from my employer. I am sure we are striving to have 0, but realistically, i have seen many that says things happen. Learn, and improve.

5 years? I remember a major outage maybe in the past year.

I believe there was a multi-hour global YouTube/Bigtable/Cloud SQL/Datastore outage in October.

Then there was the global load balancer outage in July.

Looking though the incident history, there were essentially monthly multi-region or global service disruptions of various services.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#88

Post by user5994461 »

jschwartzi wrote: post #71
izacus wrote: post #57

Earlier quoted context omitted.

It's weekend, why wouldn't you take it off? It's just silly software.

More to the point, why would you depend on Google for any critical infrastructure after this?

Because you tried to run your own infrastructure and it was so much worse.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#89

Post by eknkc »

jorams wrote: post #86
hacknat wrote: post #45

Earlier quoted context omitted.

Yeah, AWS has never had a global outage, Google has had 2 now.

I'm not sure where you get this idea, but AWS has definitely had global outages. 1.5 years ago there were massive global issues with S3, causing even their own status dashboard to malfunction. edit: I stand corrected. Apparently the S3 outage wasn't global, though its effects were. Meanwhile, this outage has only really been noticeable to ops teams, since it doesn't affect existing nodes or anything outside GKE. It's…

That only affected buckets on us-east-1.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#90

Post by owenmarshall »

eknkc wrote: post #39
scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

This might be an unpopular opinion but, Going multi region on AWS should be safe enough. If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days. You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not fee…

> You could go multi cloud though.

Multi cloud is almost always more pain than gain. You’d spend time and effort abstracting away the value that a cloud provider brings in canned services.

Hell, multi region is often more than many workloads need.

Post reply on HN