Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

291–300 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#291

Earlier quoted context omitted.

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough. Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “pri…

That does bring up an interesting point. In hindsight, we already have duplicate infrastructure - a dev account and a production account. Why in the world was it decided to put both accounts in the same region? The separate account was setup partially on my insistence but it was set up in the same region. If needed, we could have done VPC peerings across regions. ( https://aws.amazon.com/about-aws/whats-new/2017/11/a…

In most of the cases I've seen so far with multiple regions / POP (4 different companies of different sizes and verticals) the software never being deployed in another reason is the fundamental reason. Reasons ranged from "we hard-coded us-east-1 everywhere and don't know how to test region independence it turns out" to "we have 3 weeks to try to make our software distributed between different regions but have nowhere near the resources to test it well let alone develop anything new for it." Automation is completely remedial and non-repeatable in most of these companies that rarely provision clean environments from scratch. By keeping regions the same between different environments, you reduce the number of possible differences.

Some services in AWS are also not available in others (I'm quite familiar with AWS Data Pipeline not being available outside the "core" regions like us-east-1, eu-west-1) and having services in one region make usage of resources in another region is a huge change when most developers outside ones with technology literate customers are under the gun to push features out fast over sound design. The matrix of services and configurations necessary to mix and match regions and availability zones is non-trivial if you make extensive usage of AWS services above the IAAS layer.

Also, cross-region VPC peering has a TON of limitations that rather annoying depending upon how well your network has been architected (by default in most companies outside enterprises with a deep bench of network engineers, this would be rated at "complete crap barely better than a typical home wifi network"). Heck, even though I'm non-dumb at networks I have to keep reminding myself of various cross-region VPC limitations when working with refactoring cross-region VPCs like where you can reference security groups, how to propagate Route 53 records, etc.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#292

Earlier quoted context omitted.

Most feature rich cloud? I think that title belongs to AWS.

You're right in terms of breadth officially covered. But if you look at the features where they both officially have support, there are many examples where the GCP version is more reliable and usable than the AWS version. Even GKE is an example of this, despite the outage in node pool creation that we're discussing here. Way better than EKS. (Disclosure: I worked for Google, including GCP, for a few years ending in 2…

I think you're going to have to back up a claim like this with some facts.

GKE being the exception, since it was launched a couple years before EKS. AWS clearly has way more services, and the features are way deeper than GCP.

Just compare virtual machines and managed databases, AWS has about 2-3x more types of VMs (VMs with more than 4TB of RAM, FPGAs, AMD Epyc, etc.), and in databases, more than just MySQL and PostgreSQL. When you start looking at features you get features that you just can't get in GCP, like 16 read-replicas, point in time recovery, backtrack, etc.

Disclaimer: I work for AWS but my opinions are my own.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#294
post #261

Earlier quoted context omitted.

We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform. So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had ra…

I use AWS free tier and get customer support through email, but thats not the case with GCP. Do they provide free email support? If you are an early stage startup can you afford their 200/Month support, when your entire GCP bill is under $1. However, that doesn't mean you don't have to support them.

Yes, they provide using the public issue tracker. We have been used it with success.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#295
post #52

Question to Google employees: Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I’m sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich clo…

2 outages in 5 years. 5. Years. Nothing to see here, move along.

2 "global" outages. If it had been limited to a service, or a region, there would be nothing to see.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#297
post #132

Why do cloud providers have more global outages than major flagship websites like google.com?

Whey don't run on the same infra. Amazon.com doesn't run on AWS.

On the contrary, it does. They made the transition gradually.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#298

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

When everything works, GCP is the best. Stable, fast, simple, reliable. When things stop working, GCP is the worst. Slow communications and they require way too much work before escalating issues or attempting to find a solution. They already have the tools and access so most issues should take minutes for them to gather diagnostics, but instead they keep sending tickets back for "more info", inevitably followed by a…

As someone who works for Government and Enterprise - all I care about sometimes is how a company behaves when everything goes wrong.

The issue with outages for the Government organizations I have dealt with is rarely the outage itself - but strong communication about what is occurring and realistic approximate ETAs, or options around mitigation.

Being able to tell the Directors/Senior managers that issues have been "escalated" and providing regular updates are critical.

If all I could say was a "support ticket" was logged, and we are waiting on a reply (hours later) - I guarantee the conversation after the outage is going to be about moving to another solution provider with strong SLAs.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#300
post #281
post #261

Earlier quoted context omitted.

I use AWS free tier and get customer support through email, but thats not the case with GCP. Do they provide free email support? If you are an early stage startup can you afford their 200/Month support, when your entire GCP bill is under $1. However, that doesn't mean you don't have to support them.

If 200/month is an issue, then you aren’t an “early stage startup”. You’re running a hobby project.

Okay. Lets say its an hobby project, but do you understand those today's hobby projects are tomorrow's mature startup?
Post reply on HN