Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

201–210 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#201

Post by aviv »

lilbobbytables wrote: post #189
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

You're doing me a scare. I'm in the evaluation phase with them. Maybe I'm missing something here, but this is not at all what the linked post says. "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI." So, it's a UI console issue, it appears you can still manage "Affected customers can use gcloud command [1] in order to create new Node Pools. [1]" Similarly, it act…

I can't comment regarding GKE as we don't use that particular service, however we are very heavy users of many other GCP services, including Compute, Datastore, BigQuery, Pub/Sub, Storage, Functions, Speech, and others. Zero issues this weekend, everything is running 100% as any normal day.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#202

Post by james-mcelwain »

polemic wrote: post #95
shiftnight wrote: post #74

I have a question. At what point does k8s make sense? I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well. Monolith for the win! Opinions?

There's a huge range between monolith and microservice approach, and even a monolith will have dependent services. A simple web stack these days might include nginx, a database, a caching layer, some sort of task broker and then the 'monolith' web app itself. All of that can be sanely managed in k8s.

Right... IMO monolith is better understood as a reference to the data model than deployment topology. If you only have a single source of truth, then your application is naturally going to trend towards doing most of its business logic in one place. This still doesn't displace the need for other services like caching, async tasks, etc., that you identify.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#203

Post by mbrumlow »

mav3rick wrote: post #191
mbrumlow wrote: post #164

Earlier quoted context omitted.

I was so mad to read that until you said /sarcasm :p That being said I really do think there is a difference between who is working at google today and the google we all fell in love with pre-2008. I am sure there are a amazing people still working at google, but nowhere near like it was. The way I like to think about google is that some amazing people mad ea awesome train that builds tracks in front of it -- you can…

The work ethic is in tact. It is not fair to load people with stress and ask them to drop everything. You're conflating poor resource allocation with "work ethic". Burning the midnight oil when it can be avoided is not work ethic. The correct way is to load balance outage resolution.

I really don't know how to reply to you as you have set up a bunch of windmills you assumed from my previous post. Who said anything about poor resource allocation? Who said we need to load people with stress?

That being said -- when you are on call -- dropping everything is exactly what is expected.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#204

Post by dodobirdlord »

scarface74 wrote: post #103
devonkim wrote: post #80

Earlier quoted context omitted.

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough. Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “pri…

That does bring up an interesting point. In hindsight, we already have duplicate infrastructure - a dev account and a production account. Why in the world was it decided to put both accounts in the same region? The separate account was setup partially on my insistence but it was set up in the same region. If needed, we could have done VPC peerings across regions. ( https://aws.amazon.com/about-aws/whats-new/2017/11/a…

There are some significant differences region to region in AWS. Different numbers of availability zones, different latencies based on datacenter location, different types and availabilities of EC2 instances, etc. I think it makes sense to develop in the same region that your production service runs in just so you don't shoot yourself in the foot by deploying something that runs fine in your development region but doesn't run as well in your production region.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#205

Post by geggam »

cloakandswagger wrote: post #67
geggam wrote: post #30

Earlier quoted context omitted.

Infrastructure as code. Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services. This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

Totally impractical for any small business and of questionable usefulness for large ones. You'd be giving up the largest benefit of platforms like AWS--ready to use services for common tasks--to avoid the infinitesimally small chance of AWS having some doomsday global outage. To answer the original question: It looks like this issue was just a UI bug that affected the console, the service itself wasn't impacted. Even…

Evidently you dont know how to do this well

Re: Google Kubernetes Engine's third consecutive day of service disruption

#206

Post by jkaplowitz »

Jedi72 wrote: post #121
usmannk wrote: post #118

We had an issue a few weeks ago where the google front-end servers were mangling responses from Pub/Sub and returning 502 responses, making the service completely unusable and knocking over a number of things we have running in production. Despite paying for enterprise support and having in a P1 ticket, we had to spend Friday to Sunday gathering evidence to prove to the support staff that there was indeed a problem,…

They work for Google so obviously they are much smarter than you. If theres a problem its probably the customers fault. /sarcasm

Many of the tier 1 GCP support reps work for external vendors nowadays, which is probably part of the problem.

During my time on the GCE team (note I don't work at Google now) I knew multiple full-time Google employee support reps, including some still at the company. They have the good attitude and deep knowledge you'd hope for.

The problem is simply about how Google scales their GCP support org. To be completely clear, AWS support is by and large not great either.

If you're a big or strategically important customer, of course, you can get a good response from either company.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#207

Post by geggam »

scarface74 wrote: post #82
geggam wrote: post #30

Earlier quoted context omitted.

Infrastructure as code. Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services. This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

That’s not how Terraform works. Each provisioner has separate syntax depending on the cloud provider. The template for AWS wouldn’t work on GCP or Azure.

Correct. However written properly you get 90% of the way there.

Disaster recovery by switching to another provider is simple when minimal centos/rhel images are used.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#208

Post by wstrange »

Marsymars wrote: post #156
user5994461 wrote: post #100

Earlier quoted context omitted.

The baseline is that it takes 12 dedicated people across the world to run a 24/7 support operation. Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.

What's the math/logic to get to 12 people?

Timezone coverage, vacations, training, sick days, etc.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#209

Post by jkaplowitz »

scoot wrote: post #157
uluyol wrote: post #147

Earlier quoted context omitted.

Nice job cherry picking text. > 7.1 Discontinuance of Services. Subject to Section 7.2, Google may discontinue > any Services or any portion or feature for any reason at any time without > liability to Customer. Let's take a look at Section 7.2: > 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > make backwards incompatible changes to the Services specified at the URL in > the next senten…

> To me that looks like a reasonable deprecation policy. It might be, until they jack up the prices 15X with limited notice (looking at you, Google maps [1]). No deprecation needed, just force users off the platform unless they're willing to pay a massive premium. [1] https://www.google.com/search?q=google+maps+price+increase

Google Maps has never been subjected to that policy, unlike GCP services. These org chart divisions are real but only clear to Googlers, Xooglers (I'm in this category), and people who pay extremely close attention.

The fact that they're all Google makes reputation damage bleed across meaningfully different parts of what's in truth now a conglomerate under the umbrella name Google.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#210

Post by jkaplowitz »

Fizzer wrote: post #163
Jedi72 wrote: post #155

Earlier quoted context omitted.

It's ok I guess but still lets them turn it off if, in their judgement, its an economic burden i.e. costing them money. If they ever do deprecate something people have built on though they're gonna get absolutely crucified. That's probably better protection than any terms of service.

> If they ever do deprecate something people have built on though they're gonna get absolutely crucified. They do this all the time, and they get crucified every time. I built a Google Hangout App and a Chrome App, both of which were platforms eventually shut down. This is where the meme came from, and it's why I personally stopped building on top of Google products. A 1-year deprecation policy is no assurance to me…

Their approach to things like GCP is very different than their approach to those other areas of Google. But they don't separate their branding or unify their deprecation attitudes enough to avoid cross-org-chart reputation damage like this.
Post reply on HN