Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

191–200 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#191

Post by mav3rick »

mbrumlow wrote: post #164
Jedi72 wrote: post #121

Earlier quoted context omitted.

They work for Google so obviously they are much smarter than you. If theres a problem its probably the customers fault. /sarcasm

I was so mad to read that until you said /sarcasm :p That being said I really do think there is a difference between who is working at google today and the google we all fell in love with pre-2008. I am sure there are a amazing people still working at google, but nowhere near like it was. The way I like to think about google is that some amazing people mad ea awesome train that builds tracks in front of it -- you can…

The work ethic is in tact. It is not fair to load people with stress and ask them to drop everything. You're conflating poor resource allocation with "work ethic". Burning the midnight oil when it can be avoided is not work ethic. The correct way is to load balance outage resolution.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#192

Post by dodobirdlord »

jorblumesea wrote: post #153
stevehawk wrote: post #140

Earlier quoted context omitted.

This is unfortunately the norm. Like when AWS S3 went down (but couldn't update its own status images because they're in S3 and we all laughed) and along with it went Alexa, lambda, and every other service dependent on S3.

S3 is really the one of the few services on aws that can do that unfortunately. It has no concept of zone/region, it's truly global. To me it seems like a serious design flaw, as everything else in aws is striped by region, but not sure why exactly it was built like that. edit: nvm s3 has regions, it's the bucket names that are global.

Buckets are globally addressable because they planned for each S3 bucket + object key to have an associated URL (actually several), and URLs are a global namespace.

http(s)://.s3.amazonaws.com/ http(s)://s3.amazonaws.com//

Re: Google Kubernetes Engine's third consecutive day of service disruption

#193

Post by navinsylvester »

shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform.

So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had rampant impact on other services.

But i find it concerning that they provided very little update on what went wrong. I also think its better to expect nil support out of any big cloud provider if you don't have paid support. Funny how all these big cloud providers think you are not eligible for support de-facto. Sigh.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#194

Post by yardie »

kuyan wrote: post #159
Jedi72 wrote: post #155

Earlier quoted context omitted.

It's ok I guess but still lets them turn it off if, in their judgement, its an economic burden i.e. costing them money. If they ever do deprecate something people have built on though they're gonna get absolutely crucified. That's probably better protection than any terms of service.

> It's ok I guess but still lets them turn it off if, in their judgement, its an economic burden i.e. costing them money. If a service Google runs is losing money, what reason would they have to not shut it down?

This is the subtle, but important, difference between SaaS and PaaS/IaaS. Services are to use. Platforms are built upon. Flickr is a service. If they shutdown I'll get another one. If they shutdown I'll just move to another. GCP is a platform, if they shutdown I have to re-architect the entire thing from scratch.

If it's costing them money they haven't figured out a model, yet, that works in their favour.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#195

Post by dilyevsky »

shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

We had an issue a few weeks back where all nodes in west1-a could not pull docker images. Google support was pinballing P1 issue around the globe and across multiple teams for a few days untill I root caused it for them - turned out to be gce service account issues affecting entire zone. 2 days to rollback (no status page update). I know nobody gives a fuck but can’t help but feel vindicated as an ex google sre.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#196

Post by rorykoehler »

joshuamorton wrote: post #130
rorykoehler wrote: post #119

Earlier quoted context omitted.

They should take a few mins out of their weekend to update the dashboard regardless. If my small 25 employee company can do it Google can do it.

See my other comment, to say what, exactly? That's yes, it's still being investigated?

Yes precisely. At least with a new timestamp we will then know the status dashboard isn't also broken.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#197

Post by jeffcore »

davemp wrote: post #138
justinsb wrote: post #50

Earlier quoted context omitted.

I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy ( https://cloud.google.com/terms/ ), whose intent is to give you some assurances. (I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

I think it’s telling of Google’s culture that the corporate arm felt the need to formalize this in law. I won’t pretend to know what it’s telling. Just suggest that you listen for yourself. Look at rule of law versus the ideas of liberty if you’d like a stronger nudge.

AWS' Customer Agreement[1] essentially has the same language. I wouldn't be surprised to see similar language from other cloud providers as well. Seems rather prudent on their part.

[1] https://aws.amazon.com/agreement/

Re: Google Kubernetes Engine's third consecutive day of service disruption

#198

Post by rogerkirkness »

navinsylvester wrote: post #193
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform. So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had ra…

I agree with this. Compared to AWS, when Google says it's down, it's down, and that's rare. When they say it's up, it's up.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#199

Post by doodliego »

joshuamorton wrote: post #128
michaelt wrote: post #120

Earlier quoted context omitted.

At a smaller, less corporate company the engineer/public comms person dividing line would not be so ossified the divide couldn't be bridged when the situation called for it.

I think the broader point here is that they (whomever they is) don't think the situation calls for it. That's why they said the next update would be on Monday on Friday. You may feel that's a bad decision, but I doubt that people are in a panic because they can't push out an update that would not be noticably different from the last one. Just to clarify, what should this update contain?

Compare Amazon's responses during their S3 outage: https://www.theregister.co.uk/2017/03/01/aws_s3_outage/ (toward the bottom)

What's the better communication plan: detailed, hourly updates or terse, one-line blog posts scattered across several days?

Re: Google Kubernetes Engine's third consecutive day of service disruption

#200

Post by dilyevsky »

justinsb wrote: post #29

Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now! It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

So, given that i filed this months ago via official support and it’s still not fixed, can you look into misleading container memory reporting ui bug. It reports memory_total but should be working_set
Post reply on HN