Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

blog.jetstack.io

Post reply on HN
Print view View original

51–58 of 58 posts

Re: Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

#51

Post by rossmohax »

solatic wrote: post #25
rossmohax wrote: post #6

> This means the Service must be created first, then when the corresponding NEG is created the name can be queried and added to the Terraform project. These kind of reverse dependencies, where app level changes have to be reflected in infrastructure are source of endless bugs and headaches. To do it right, you'd want to encode app and infra changes in the same ubiquitous tool, but "infrastructure as a code" tools suc…

> To do it right, you'd want to encode app and infra changes in the same ubiquitous tool Well, that's exactly what we do with Dhall, which lets us put static types on infrastructure tooling that prints out unpredictable output but in a predictable format and transform that output into the format that other dependent tools expect. Everything is then glued together with simple shell scripts and run in a CI environment.…

I looked at Dhall before. It is a pure language for static configs, right? It can't help me to express "deploy k8s resource, wait for annotation to appear, take annotation value and use it to deploy infra resource". Wrapping multiple tools in a bash is exact thing I wish I didn't have to do :(

If Dhall enables safer bash glue, I'd be happy to read how

Re: Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

#52

Post by ec109685 »

manigandham wrote: post #49
ec109685 wrote: post #48

Earlier quoted context omitted.

Deploying an app to more than one geographic region on earth should be very basic. For mostly read only sites, it is mean to make your users suffer extra latency and your reliability goes down. The authors had to go read source code to figure out obscure annotations at one point to get container aware networking to work, which then wasn’t compatible with Google’s tools to setup global ingress.

That isn't very basic and never has been. If it's just a read-only site then all you need is a CDN, not load balancing with multiple regions running Kubernetes clusters. What other cloud vendor or PaaS offers this? The only one that has seamless global deployment is Cloudflare's Workers but that's limited to Javascript code in a lightweight functions environment.

I said mostly read-only. Like if you can run off of a MySQL master and replica.

It’s common enough they are trying to support it: https://cloud.google.com/blog/products/gcp/how-to-deploy-geo...

Their tool is at least three years old and still has this disclaimer: “Caution: The kubemci tool is a temporary solution intended to help users begin using multi-cluster Ingress. This tool will be replaced by an implementation using kubectl that delivers a more Kubernetes-native experience. Once the kubectl implementation is available, you will need to manually migrate any apps that use kubemci”

Re: Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

#53

Post by hinkley »

rossmohax wrote: post #50
hinkley wrote: post #43

Earlier quoted context omitted.

What about the Pulumi solution gets in your way?

No tools IaC tools I know of including Pulumi, allow to describe evolutions for infra. Simple use case - deploy canary instance, check it's health, scale down canary instance, update main pool of instances. In other words, there is no way to express sequence of changes of the same object in the single run of IaC tool. We are forced to wrap IaC tools in layers of bash to simulate it.

Ah, good to know, thanks.

What you're describing sounds a bit like migrations for databases. Do you agree?

Re: Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

#54

Post by whatsmyusername »

atombender wrote: post #44
whatsmyusername wrote: post #37

Earlier quoted context omitted.

I've used k8s at various times since it's inception. I've yet to see a compelling argument for running your control plane on the same layer as your workloads. I have always found this to be a recipe of disaster when 'something' in your service dependency chain breaks. Also not having load balancer support built in kills it for me. Yeah you can do metallb or nginx ingress but it means they punted one of the major comp…

Kubernetes splits the control plane from the workloads. You have to run Kubelet on the individual nodes, but it's in charge of node-local orchestration; sure, you could maybe invent something thinner that was controlled with just with SSH or something, but you can't really get away with some kind of controller on the node itself. Something has to start containers and monitor their status, for one. There are lots of p…

Yeah I've used traefik (originally on Mesos though on Kubernetes as well). I mean it's okay, but I don't get security groups and I have to build stuff. I'm old. If you make me build commodity pieces and your competitor doesn't I'm going with the competitor.

ECS with Fargate behind ALBs talking to RDS/SQS/Elasticache with Scheduled Tasks as my cron layer is 99% of what we need without standing up a single host we have to maintain.

I put that in my calculator and it makes a happy face.

Re: Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

#55

Post by whatsmyusername »

minxomat wrote: post #27
whatsmyusername wrote: post #22

I've yet to see a compelling argument for the complication of kubernetes or the 'its not AWS' nature of GCP over a bog standard AWS ALB with ECS cluster tasks attached to it.

Because k8s mostly prevents lock-in. A much better idea is to run EKS on Fargate. You still won't have to manage a cluster, but at least you can now use standard k8s manifests that work in any other cluster instead of the ECS homebrew stuff. You can keep ALB, because ALB can run as an Ingress Controller.

EKS billing is hilariously bad (Managed SFTP is the same way). It's obvious they built it as a feature parity thing and they don't want you to actually use it.

I can run all of my container infrastructure for a property for the cost of the control plane for EKS (before you attach hosts and before you spend how much time figuring out why hosts won't attach to it). It's just bad.

Re: Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

#56

Post by auspex »

whatsmyusername wrote: post #54
atombender wrote: post #44

Earlier quoted context omitted.

Kubernetes splits the control plane from the workloads. You have to run Kubelet on the individual nodes, but it's in charge of node-local orchestration; sure, you could maybe invent something thinner that was controlled with just with SSH or something, but you can't really get away with some kind of controller on the node itself. Something has to start containers and monitor their status, for one. There are lots of p…

Yeah I've used traefik (originally on Mesos though on Kubernetes as well). I mean it's okay, but I don't get security groups and I have to build stuff. I'm old. If you make me build commodity pieces and your competitor doesn't I'm going with the competitor. ECS with Fargate behind ALBs talking to RDS/SQS/Elasticache with Scheduled Tasks as my cron layer is 99% of what we need without standing up a single host we have…

Fargate is actually one of the most expensive ways to run a container from a pure cloud cost standpoint.

Re: Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

#57

Post by whatsmyusername »

auspex wrote: post #56
whatsmyusername wrote: post #54

Earlier quoted context omitted.

Yeah I've used traefik (originally on Mesos though on Kubernetes as well). I mean it's okay, but I don't get security groups and I have to build stuff. I'm old. If you make me build commodity pieces and your competitor doesn't I'm going with the competitor. ECS with Fargate behind ALBs talking to RDS/SQS/Elasticache with Scheduled Tasks as my cron layer is 99% of what we need without standing up a single host we have…

Fargate is actually one of the most expensive ways to run a container from a pure cloud cost standpoint.

Not if you take into account the fact that I don't have to maintain the hosts the jobs are running on (which is a not inconsiderable amount of time when you get HIPAA and PCI into the mix). Liberal application of AWS Savings Plans with 1 year no up front helps a lot too.

At scale I agree, however if you're a small company (or, in our case, 6 small companies) and don't have dedicated DevOps (or, in our case, have 1.5 DevOps people split 6 ways), it's fantastic.

Re: Container-Native Multi-Cluster Global Load Balancing with Cloud Armor on GCP

#58

Post by ec109685 »

manigandham wrote: post #49
ec109685 wrote: post #48

Earlier quoted context omitted.

Deploying an app to more than one geographic region on earth should be very basic. For mostly read only sites, it is mean to make your users suffer extra latency and your reliability goes down. The authors had to go read source code to figure out obscure annotations at one point to get container aware networking to work, which then wasn’t compatible with Google’s tools to setup global ingress.

That isn't very basic and never has been. If it's just a read-only site then all you need is a CDN, not load balancing with multiple regions running Kubernetes clusters. What other cloud vendor or PaaS offers this? The only one that has seamless global deployment is Cloudflare's Workers but that's limited to Javascript code in a lightweight functions environment.

They are making one part of this easier at least: https://twitter.com/vicnastea/status/1232751949117702145?s=2...
Post reply on HN