Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

241–250 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#241

Post by omeid2 »

brian-armstrong wrote: post #129
justinsb wrote: post #50

Earlier quoted context omitted.

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy ( https://cloud.google.com/terms/ ), whose intent is to give you some assurances. (I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

What happens when they suddenly deprecate the deprecation policy?

Nothing, sort of. Subject to "Section 1.7 Modifications" of Terms:

b. To the Agreement:

Google may make changes to this Agreement, including pricing (and any linked documents) from time to time. .... Google will provide at least 90 days’ advance notice for materially adverse changes to any SLAs by either: (i) sending an email to Customer’s primary point of contact; (ii) posting a notice in the Admin Console; or (iii) posting a notice to the applicable SLA webpage. If Customer does not agree to the revised Agreement, please stop using the Services. Google will post any modification to this Agreement to the Terms URL.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#242

Post by omeid2 »

uluyol wrote: post #147
chrisseaton wrote: post #137

Earlier quoted context omitted.

> Google may discontinue any Services or any portion or feature for any reason at any time without liability to Customer for any reason at any time

Nice job cherry picking text. > 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue > any Services or any portion or feature for any reason at any time without > liability to Customer. Let's take a look at Section 7.2: > 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > make backwards incompatible changes to the Services specified at the URL in > the next senten…

Let's not forget that Google can change the terms as they please with a 90 Days notice as per Section 7.1 Modifications of the terms. So any promise that is longer than 90 days, even without a escape hatch like Section 7.2, would be legally weak and subject to change at any time without much recourse.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#243

Post by timdumol »

lilbobbytables wrote: post #189
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

You're doing me a scare. I'm in the evaluation phase with them. Maybe I'm missing something here, but this is not at all what the linked post says. "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI." So, it's a UI console issue, it appears you can still manage "Affected customers can use gcloud command [1] in order to create new Node Pools. [1]" Similarly, it act…

We've had no issues deleting and creating node pools this weekend (on asia-east1-a). No other problems noticed either.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#244

Post by halbritt »

scarface74 wrote: post #218

Say I were a CTO (I’m nowhere near it), why would I choose GCP over AWS or Azure? Even if after doing a technical assessment and I thought that GCP was technically slightly better, if something happened, the first question I would be asked is “why did you choose GCP over AWS?” No one would ever ask why you chose AWS. The old “no one ever got fired for buying IBM”. Even if you chose Azure because you’re a Microsoft sh…

GCP has a few features that set it apart from other cloud providers. GKE is head and shoulders above the other offerings from AWS and Azure.

GCP can be a fair bit cheaper than AWS and Azure for certain workloads. Raw compute/memory is about the same. Storage can make a big difference. GCP persistent SSD costs a bit more than AWS GP2 with much better performance and way cheaper than IO2. Local SSD is also way, way cheaper than I2 instances.

Most folks deploying distributed data stores that need guaranteed performance are using local disk, so this can be a really big deal.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#245

Post by lstamour »

sladey wrote: post #24

Seems to be some weird underlying issue going on at GCP at the moment. Had cloud build webhooks returning a 500 error. Noticed we were at 255 images and deleting some fixed the issue. Created a P2 ticket about the issue before we managed to solve it and haven't had a response in 40+ hours. The timeline of this disruption matches when we started experiencing cloud build errors.

Outsider here, but I believe Cloud Build runs on GKE Jobs, so if they’re having trouble, it does indeed sound related.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#246

Post by halbritt »

rlancer wrote: post #174

UPDATE: Got some clarity, these issues are caused by "resource exhaustion" meaning there are no resources left to be allocated.

I'm curious to see if this is true.

I faced some pretty serious resource allocation issues earlier in the year. The us-west1-a region was oversubscribed. I was unable to get any real information from support with regard to capacity. Eventually my rep gave me some qualitative information that I was able to act on.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#247

Post by halbritt »

owenmarshall wrote: post #90
eknkc wrote: post #39

Earlier quoted context omitted.

This might be an unpopular opinion but, Going multi region on AWS should be safe enough. If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days. You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not fee…

> You could go multi cloud though. Multi cloud is almost always more pain than gain. You’d spend time and effort abstracting away the value that a cloud provider brings in canned services. Hell, multi region is often more than many workloads need.

My entire infrastructure is on k8s which should make multi-cloud easy...

Nope.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#248

Post by jorblumesea »

dodobirdlord wrote: post #192
jorblumesea wrote: post #153

Earlier quoted context omitted.

S3 is really the one of the few services on aws that can do that unfortunately. It has no concept of zone/region, it's truly global. To me it seems like a serious design flaw, as everything else in aws is striped by region, but not sure why exactly it was built like that. edit: nvm s3 has regions, it's the bucket names that are global.

Buckets are globally addressable because they planned for each S3 bucket + object key to have an associated URL (actually several), and URLs are a global namespace. http(s):// .s3.amazonaws.com/ http(s)://s3.amazonaws.com/ /

urls would have to be global, but why buckets themselves? It seems like a many to one relationship would easily be possible.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#249

Post by halbritt »

scarface74 wrote: post #23

A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally. What would a small business do as a contingency plan?

My entire production infrastructure is in GCP. What happened here has caused approximately zero impact to the availability of my service.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#250

Post by 013a »

scarface74 wrote: post #218

Say I were a CTO (I’m nowhere near it), why would I choose GCP over AWS or Azure? Even if after doing a technical assessment and I thought that GCP was technically slightly better, if something happened, the first question I would be asked is “why did you choose GCP over AWS?” No one would ever ask why you chose AWS. The old “no one ever got fired for buying IBM”. Even if you chose Azure because you’re a Microsoft sh…

- Native integration with G-Suite as an identity provider. Unified permissions modeling from the IDP, to work apps like email/Drive, to cloud resources, all the way into Kubernetes IAM.

- Security posture. Project Zero is class leading, and there's absolutely a "fear-based" component there, with the open question of when Project Zero discovers a new exploit, who will they share it with before going public? The upcoming Security Command Center product looks miles ahead of the disparate and poorly integrated solutions AWS or Azure offers.

- Cost. Apples to apples, GCP is cheaper than any other cloud platform. Combine that with easy-to-use models like preemptible instances which can reduce costs further; deploying a similar strategy to AWS takes substantially more engineering effort.

- Class leading software talent. Google is proven to be on the forefront of new CS research, then pivoting that into products that software companies depend on; you can look all the way back to BigQuery, their AI work, or more recently in Spanner or Kubernetes.

- GKE. Its miles ahead of the competition. If you're on Kubernetes and its not on GKE, then you've got legacy reasons for being where you're at.

Plenty of great reasons. Reliability is just one factor in the equation, and GCP definitely isn't that far behind AWS. We have really short memories as humans, but too soon we seem to forget Azure's global outage just a couple months ago due to a weather issue at one datacenter, or AWS's massive us-east-1 S3 outage caused by a human incorrectly entering a command. Shit happens, and it's alright. As humans, we're all learning, and as long as we learn from this and we get better then that's what matters.

Post reply on HN