Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine's third consecutive day of service disruption

status.cloud.google.com

Post reply on HN
Print view View original

261–270 of 419 posts

Re: Google Kubernetes Engine's third consecutive day of service disruption

#261

Post by vira28 »

navinsylvester wrote: post #193
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

We are GCP customers for the last couple of years. We use other cloud platforms(AWS, IBM, Oracle, OrionVM) too. We don't use GKE but use rancher/kubernetes combo on their standard platform. So far GCP is the best, hands down in terms of stability. We never had a single outage or maintenance downtime notification till now. We are power users but our monitoring didn't pick any anomaly so i don't think this issue had ra…

I use AWS free tier and get customer support through email, but thats not the case with GCP. Do they provide free email support?

If you are an early stage startup can you afford their 200/Month support, when your entire GCP bill is under $1. However, that doesn't mean you don't have to support them.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#262

Post by bg24 »

nstart wrote: post #258
shiftnight wrote: post #74

I have a question. At what point does k8s make sense? I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well. Monolith for the win! Opinions?

As someone whose daily work happens on k8s, I'd say you better be paining a lot before you move to k8s. I take great care to avoid this, but if you aren't careful, you can end up "feeling" productive on k8s without actually being productive. K8s gives a lot of room for one to tweak workflows, discuss deployment strategies, security, "best practices", etc. And you can get things done reasonably fast. But that's like a…

Very nicely written. While not a direct response to OP, you articulated some great points on k8s. k8s will naturally succeed as the future of data center orchestration as VM's give way to containers. But it is questionable if everyone needs it.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#263

Post by 013a »

shiftnight wrote: post #74

I have a question. At what point does k8s make sense? I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well. Monolith for the win! Opinions?

I hate the word Microservice, so I'm just going to use the word Service.

Most monoliths software companies build aren't actually monoliths, conceptually. Let's say you integrate with the Facebook API to pull some user data. Facebook is, within the conceptual model of your application, a service. Hell, you even have to worry "a little bit" about maintaining it; provisioning and rotating API keys, possibly paying for it, keeping up to date on deprecations, writing code to wire it up, worrying about network faults and uptime... That sounds like a service to me; we're three steps short of a true in-house service, as you don't have to worry about writing its code and actually running it, but conceptually its strikingly similar.

Facebook is a bad example here. Let's talk Authentication. Its a natural "first demonolithized service" that many companies will reach to build. Auth0, Okta, etc will sell you a SaaS product, or you can build your own with many freely available libraries. Conceptually they fill the same role in your application.

Let's say you use Postgres. That's pretty much a service in your application. A-ha; that's a cool monolith you've got there, already communicating over a network ain't it. Got a redis cache? Elasticsearch? Nginx proxy? Load balancer? Central logging and monitoring? Uh oh, this isn't really looking like a monolith anymore is it? You wanted it to be a monolith, but you've already got a few networked services. Whoops.

"Service-oriented" isn't first-and-foremost a way of building your application. It's a way of thinking about your architecture. It means things like decoupling, gracefully handling network failures, scaling out instead of up, etc. All of these concepts apply whether you're building a dozen services or you're buying a dozen services.

Monolithic architectures are old news because of this recognition; no one builds monoliths anymore. It's arguable if anyone ever did, truly. We all depend on networked services, many that other people provide. The sooner you think in terms of networked services, the sooner your application will be more reliable and offer a superior experience to customers.

And then, it's a natural step to building some in-house. I am staunchly in the camp of "'monolith' first, with the intention of going into services" because it forces you to start thinking about these big networking problems early. You can't avoid it.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#264

Post by scarface74 »

013a wrote: post #250
scarface74 wrote: post #218

Say I were a CTO (I’m nowhere near it), why would I choose GCP over AWS or Azure? Even if after doing a technical assessment and I thought that GCP was technically slightly better, if something happened, the first question I would be asked is “why did you choose GCP over AWS?” No one would ever ask why you chose AWS. The old “no one ever got fired for buying IBM”. Even if you chose Azure because you’re a Microsoft sh…

- Native integration with G-Suite as an identity provider. Unified permissions modeling from the IDP, to work apps like email/Drive, to cloud resources, all the way into Kubernetes IAM. - Security posture. Project Zero is class leading, and there's absolutely a "fear-based" component there, with the open question of when Project Zero discovers a new exploit, who will they share it with before going public? The upcomi…

Your response is from a geek’s viewpoint. No insult, intended, I’m first and foremost a 30 year computer geek myself - started programming in 65C02 assembly in 6th grade and still mostly hands on.

But, whether it is right or not, as an architect/manager, etc, you have to think about what’s not just best technically. You also have to manage your reputational risks if things go south and less selfishly, how quickly can you find someone with the relevant experience.

From a reputation standpoint, even if AWS and GCP have the same reliability, no one will blame you if AWS goes down if you followed best practices. If a global outage of an AWS resource went down, you’re in the same boat as a ton of other people. If everyone else was up and running fine but you weren’t because you were on the distant third cloud provider, you don’t have as much coverage.

I went out on a limb and chose Hashicorp’s Nomad as the basis of a make or break my job project I was the Dev lead/architect for hoping like hell things didn’t go south and the first thing people were going to ask me is why I chose it. No one had heard of Nomad but I needed a “distributed cron” type system that could run anything and it was on prem. It was the right decision but I took a chance.

From a staffing standpoint, you can throw a brick and hit someone who at least thinks they know something about AWS or Azure GCP, not so much.

It’s not about which company is technically better, but I didn’t want to ignore your technical arguments...

Native integration with G-Suite as an identity provider. Unified permissions modeling from the IDP, to work apps like email/Drive, to cloud resources, all the way into Kubernetes IAM.

You can also do this with AWS - use a third party identity provider and map them to native IAM user and roles.

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_cr...

Cost. Apples to apples, GCP is cheaper than any other cloud platform. Combine that with easy-to-use models like preemptible instances which can reduce costs further; deploying a similar strategy to AWS takes substantially more engineering effort.

The equivalent would be spot instances on AWS.

From what (little) I know about preemptible instances, it seems kind of random when they get reassigned but Google tries to be fair about it. The analagous thing on AWS would be spot instances where you set the amount you want to pay.

Class leading software talent. Google is proven to be on the forefront of new CS research, then pivoting that into products that software companies depend on; you can look all the way back to BigQuery, their AI work, or more recently in Spanner or Kubernetes.

All of the cloud providers have managed Kubernetes.

As far as BigQuery. The equivalent would be Redshift.

https://blog.panoply.io/a-full-comparison-of-redshift-and-bi...

Reliability is just one factor in the equation, and GCP definitely isn't that far behind AWS

Things happen. I never made an argument about reliability.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#265

Post by scarface74 »

halbritt wrote: post #244
scarface74 wrote: post #218

Say I were a CTO (I’m nowhere near it), why would I choose GCP over AWS or Azure? Even if after doing a technical assessment and I thought that GCP was technically slightly better, if something happened, the first question I would be asked is “why did you choose GCP over AWS?” No one would ever ask why you chose AWS. The old “no one ever got fired for buying IBM”. Even if you chose Azure because you’re a Microsoft sh…

GCP has a few features that set it apart from other cloud providers. GKE is head and shoulders above the other offerings from AWS and Azure. GCP can be a fair bit cheaper than AWS and Azure for certain workloads. Raw compute/memory is about the same. Storage can make a big difference. GCP persistent SSD costs a bit more than AWS GP2 with much better performance and way cheaper than IO2. Local SSD is also way, way che…

I have a more detailed post above, but if you are large enough, you’re not paying the listed price for AWS. But even if you are, prices change all of the time. From a completely selfish standpoint, is the price difference worth the cost to bet your reputation on if you are the one that made the final decision? Even if statistically the same could happen with AWS, no one would blame you for choosing AWS.

However, I could see doing a multicloud solution where I took advantage of the price difference for one project.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#266

Post by shareometry »

lilbobbytables wrote: post #189
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

You're doing me a scare. I'm in the evaluation phase with them. Maybe I'm missing something here, but this is not at all what the linked post says. "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI." So, it's a UI console issue, it appears you can still manage "Affected customers can use gcloud command [1] in order to create new Node Pools. [1]" Similarly, it act…

You are right about the Google blog content itself not indicating three days of outage. Turns out they just forgot to mark that particular issue as resolved on Friday, as you point out. This is my mistake. I would update my comment to reflect this, but it doesn't seem to allow an edit at this point.

The items I put down in my comment are based largely on user reports, though (there isn't much else to go on). And I mean these items as questions (i.e. "is this accurate?"). Folks here on HN have definitely been reporting ongoing problems and seem to be suggesting that they are not resolved and are actually larger in scope than the Google blog post addressed.

Someone from Google commented here a few hours ago indicating Google was looking into it. And other folks here are reporting that they don't have the same problems. So it's kind of an open question what's going on.

I'm in the evaluation phase too. And I've found a lot to like about GCP. I'm hoping the problems are understandable.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#267

Post by chimen »

scarface74 wrote: post #218

Say I were a CTO (I’m nowhere near it), why would I choose GCP over AWS or Azure? Even if after doing a technical assessment and I thought that GCP was technically slightly better, if something happened, the first question I would be asked is “why did you choose GCP over AWS?” No one would ever ask why you chose AWS. The old “no one ever got fired for buying IBM”. Even if you chose Azure because you’re a Microsoft sh…

I chose gcloud for the easier admin interface. I somehow manage to separate all my resources and look at them based on project groups without having to know cryptic instance ids of 32 chars. Oh, and they had Kubernetes first and I jumped into that boat early.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#268

Post by raincom »

lilbobbytables wrote: post #189
shareometry wrote: post #150

I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly: 1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN). 2) This issue was global in scope, affecti…

You're doing me a scare. I'm in the evaluation phase with them. Maybe I'm missing something here, but this is not at all what the linked post says. "We are investigating an issue with Google Kubernetes Engine node pool creation through Cloud Console UI." So, it's a UI console issue, it appears you can still manage "Affected customers can use gcloud command [1] in order to create new Node Pools. [1]" Similarly, it act…

If you run your own k8s on GCP, you are not going to be affected by GKE.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#269

Post by pfd1986 »

antpls wrote: post #104
carbocation wrote: post #99

Earlier quoted context omitted.

> I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west. I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA? I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).

Tried on us-east, us-north, europe-west, also tried asia, with different instance sizes and with both UI and CLI. None worked for me.

Same here.

Re: Google Kubernetes Engine's third consecutive day of service disruption

#270

Post by scarface74 »

chimen wrote: post #267
scarface74 wrote: post #218

Say I were a CTO (I’m nowhere near it), why would I choose GCP over AWS or Azure? Even if after doing a technical assessment and I thought that GCP was technically slightly better, if something happened, the first question I would be asked is “why did you choose GCP over AWS?” No one would ever ask why you chose AWS. The old “no one ever got fired for buying IBM”. Even if you chose Azure because you’re a Microsoft sh…

I chose gcloud for the easier admin interface. I somehow manage to separate all my resources and look at them based on project groups without having to know cryptic instance ids of 32 chars. Oh, and they had Kubernetes first and I jumped into that boat early.

Were you making that decision for a personal project or for something work related?

The AWS console is wildly inconsistent. I’ll give you that. But, any projects I am doing are usually defined by a Cloud Formation Template abd I can see all of the related resources by looking at the stack that was run.

Theoretically, you could use the stack price estimator, I haven’t tried it though.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...

Post reply on HN