Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

231–240 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#231
post #207

Earlier quoted context omitted.

That’s not how Terraform works. Each provisioner has separate syntax depending on the cloud provider. The template for AWS wouldn’t work on GCP or Azure.

Correct. However written properly you get 90% of the way there. Disaster recovery by switching to another provider is simple when minimal centos/rhel images are used.

Then struggle like hell for the other 90%

Re: Google Kubernetes Engine's third consecutive day of service disruption

#232
post #31

Earlier quoted context omitted.

Assuming you have architected multi-region I'm not sure how realistic that scenario is. AWS regions are mostly standalone, I have seen services go down in a region but never globally.

The most recent example that I can think of when something went down globally was Route 53 - the one service that AWS promises 100% up time for.

Citation needed?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#233

Earlier quoted context omitted.

> When an entire region is down what I have noticed is that all things are fucked globally on aws. Do you have an example on this?

Just grabbed first article. Example: In this case capitalone went down. I don’t work at capitalone - but I imagine they had their data copied across every region 30 times. https://www.geekwire.com/2018/widespread-outage-amazon-web-s...

I think you're much too optimistic about capitalone. They probably had a single point of failure, possibly one they didn't realize they had.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#234

Earlier quoted context omitted.

“2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.” Ok. So on aws we were* paying for putting systems across regions, but, honestly I don’t get the point. When an entire region is down what I have noticed is that all things are…

> When an entire region is down what I have noticed is that all things are fucked globally on aws. Do you have an example on this?

On 17 October, there was a multi-AZ network failure at us-east-1. It only lasted 3m35s, but it was enough that our customers were calling about our site being down.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#235
post #207

Earlier quoted context omitted.

That’s not how Terraform works. Each provisioner has separate syntax depending on the cloud provider. The template for AWS wouldn’t work on GCP or Azure.

Correct. However written properly you get 90% of the way there. Disaster recovery by switching to another provider is simple when minimal centos/rhel images are used.

Assume I’m not fully informed here. What does “written properly” mean? Sure I can move over route53 to cloud DNS easily but Firehose to PubSub? Lambda to Cloud Functions? DynamoDB to BigTable and moving the data?

The syntax for provisioning these doesn’t work that well for some find and replace to work. Are you using a templater to generate cloud-specific HCL from a template or something? Sounds like a pretty big problem to solve to me and not just something where you can win via discipline.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#236

Earlier quoted context omitted.

A lot of people's businesses, reputations and livelihoods depend on "silly software," not to mention that they are paying customers themselves.

I know sarcasm doesn't translate well on the internet, but how the heck did you manage to miss it in OPs post?!

Ah, I guess I missed it. My fault.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#237

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

This is why GCP has no hope of ever taking significant market share from AWS. Google thinks they can treat their cloud customers like they treat users of their free services. Customer support and communication are essential.

I'm not sure about the market share, but I agree with the last two sentences.

...and I'm a happy GCP customer.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#238
post #30

Earlier quoted context omitted.

Infrastructure as code. Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services. This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

I think you are downplaying minimal

No, with over 20 years experience and having done exactly this for several different companies and startups I pretty much have this process down.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#239

We had an issue a few weeks ago where the google front-end servers were mangling responses from Pub/Sub and returning 502 responses, making the service completely unusable and knocking over a number of things we have running in production. Despite paying for enterprise support and having in a P1 ticket, we had to spend Friday to Sunday gathering evidence to prove to the support staff that there was indeed a problem,…

Did they ask you for a screenshot?

That's my favorite.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#240
post #121

Earlier quoted context omitted.

They work for Google so obviously they are much smarter than you. If theres a problem its probably the customers fault. /sarcasm

Many of the tier 1 GCP support reps work for external vendors nowadays, which is probably part of the problem. During my time on the GCE team (note I don't work at Google now) I knew multiple full-time Google employee support reps, including some still at the company. They have the good attitude and deep knowledge you'd hope for. The problem is simply about how Google scales their GCP support org. To be completely cl…

80% of my support experiences are laughably bad.

20% of my support experiences are amazing.

Fortunately, I don't require decent support to keep my service running. My sales rep tells me that he's aware of the problem.

I speculate it's simply the result of GCP trying to grow the org very quickly.

Post reply on HN