Google Kubernetes Engine is introducing a cluster management fee on June 6 • Hacker News Board

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#451

Post by hitpointdrew » Thu, Mar 05, 2020, 1:02 AM UTC

lvh wrote: post #86

rolleiflex wrote: post #79

Earlier quoted context omitted.

We’re in the same situation — we’ve engineered for minimum provider-specific dependencies but GKE LoadBalancers were where they got us via arm twisting as well. There is no way to expose a cluster to the outside world in a production environment otherwise.

It's kind of ridiculous internal load balancers can't get automatic certs. We've had to do a stupid dance just to get certs via the LE DNS challenge out of band, and then regularly install them on internal LBs.

Maybe I don't understand your problem, but can't you just use Traefik (https://docs.traefik.io/user-guides/crd-acme/). It will get certs from letsencrypt for you.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#452

Post by apple4ever » Thu, Mar 05, 2020, 1:03 AM UTC

p_l wrote: post #405

sethvargo wrote: post #129

Earlier quoted context omitted.

Thank you These changes won't take effect until June - customers won't start getting billed immediately. I'm sorry that you feel trapped, that's not our intention. > You should keep existing clusters in the pricing model they’ve been built in, and apply this change for clusters created after today. This is great feedback, but clusters should be treated like cattle, not pets. I'd love to learn more about why your clus…

> This is great feedback, but clusters should be treated like cattle, not pets. I'd love to learn more about why your clusters must be so static. Clusters often are not "cattle". If your operation is big enough, then yes, they might be. Usually they aren't, they are named systems and represent mostly static entity, even if the components of said entity change every hour. Personally, I'm running in production a cluste…

I think our architects’ heads would explode if they were told we should treat them like cattle.

For us, Clusters are a promise to our developers. We can’t just spin up a new cluster because we feel like it. I must be missing something or maybe our culture is just different.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#453

Post by hitpointdrew » Thu, Mar 05, 2020, 1:05 AM UTC

bcheung wrote: post #364

pojzon wrote: post #107

Earlier quoted context omitted.

How about managing own k8s running on VMs / bare-metal? Pretty much anyone who works in ops longer understood from the go that its impossible to be totally provider-agnostic. K8S is just a nice api on top of provider api that still requires provider specific configuration.

I'm running my own on bare metal dedicated servers. You will need to install a few extra things (MetalLB for LoadBalancer, CertManager for SSL, an ingress controller (nginx, Ambassador, Gloo), and one of the CSI plugins for your preferred storage method). It is extra work but as a personal cluster for hobby work, I'm paying $65/mo total for the cluster. Same specs would probably be $1000/mo at a public cloud provider…

Traefik can also automatically get letsencrypt certs, if you don't want to use CertManager. Traefik gives the added benefit of also doing Ingress.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#454

Post by busterarm » Thu, Mar 05, 2020, 1:06 AM UTC

btmiller wrote: post #414

rolleiflex wrote: post #160

Earlier quoted context omitted.

We use Skaffold and it’s great. I’m talking about very minor unforeseen stuff that causes outages, not that we do it manually.

This is an interesting exchange if only for the thread developing instead of a single reply from a rep; it’s nice to see that level of engagement. More importantly, this dialogue speaks volumes to Google’s stubbornness. Seth’s/Google’s position is: do it the Google way, sorry-not-sorry to all those that don’t fit into our model. Like we haven’t heard of infrastructure as code? That can’t paper over basics like being…

Google stubbornly resists the idea that their platforms have actual users who depend on things not being broken for them constantly. It's cultural.

AWS has the complete opposite model.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#455

Post by jjoonathan » Thu, Mar 05, 2020, 1:09 AM UTC

thockingoog wrote: post #228

alasdair_ wrote: post #218

Earlier quoted context omitted.

>If the main value of GKE over DIY is $73, you should totally DIY. It's not the fee itself, it's the worry that GKE will do what Google Maps did and massively increase fees with very little notice, causing people to scramble to migrate. Google has a really bad reputation right now when it comes to cancelling projects that people have built their businesses upon, or jacking up fees quickly. The $73 is irrelevant on it…

Google and Google Cloud are largely different businesses, though I understand it's hard to keep that in mind in the context of things like this. I encourage everyone to always stay nimble and keep your eyes on portability. I also encourage you to try to assess the REAL costs of doing things yourself. It's rarely as cheap as you think it is. As a Kubernetes maintainer, I am fanatical about portability. As part of the…

If you brand it as "Google," you can't expect the positive associations to transfer and the negative associations not to transfer. That's not how it works. You get both or you get neither.

Keep in mind that you aren't selling to me, you are selling to middle management who hears "it's different we promise" and then goes home to have nightmares about their rival manager piling on: "GCP canceled a service from under you? Who could possibly have seen that coming? Oh, the salesman told you it wouldn't happen, my mistake. (Everyone laughs at manager's stupidity.)"

GCP needs to give these guys ammunition. AWS burns goodwill like they've got a city to light: surprise bills, abandoned (but technically not canceled) services, poor performance, sticky abstractions, shameless grand announcements of services that upon further investigation only exist in the sense that you can ask support and wait 5 days (rDNS), etc. Broken software and subtle (or overt!) killer caveats abound. Go with AWS, we scale to the moon! Oh, "the moon" is >5GB of data through our time series ML? Lol no. Hard limit. Oh, we let your buddies exceed that limit and we're advertising that you can exceed the limit? Lol -- not our problem. Yet nobody holds them accountable. Nobody gets fired for choosing AWS, because AWS over-promising and under-delivering is not a meme. It's reality, especially by Google standards, so GCP marketing could make it a meme if they had any sense about them, but as far as I can tell they aren't interested.

If GCP stays the course, the results are 100% predictable and frustrating as hell, because AWS is a real steaming pile and I hate to see them continue to win The Game because Google, of all companies, can't figure out how to advertise and manage their reputation.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#456

Post by apple4ever » Thu, Mar 05, 2020, 1:11 AM UTC

freedomben wrote: post #170

takeda wrote: post #131

Earlier quoted context omitted.

From what I've seen looks like managing k8s on your own often ends up requiring a dedicated team to keep with their insane release cycle.

Can confirm. Depending on your cluster size you will need at least 2 dedicated people on the "Kubernetes" team. You'll probably also end up rolling-your-own deployment tools because K8s API is a little overwhelming for most devs.

Confirmed here too. I don’t think management realized the amount of toil that Kube requests to stay up to date.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#457

Post by manigandham » Thu, Mar 05, 2020, 1:14 AM UTC

ones_and_zeros wrote: post #175

rolleiflex wrote: post #90

Earlier quoted context omitted.

Seth — I appreciate you being here to take feedback, and for the clarification as well. The very surprising email I’ve received this morning is very hazy on the details, and the docs linked from the email are not updated yet. The main issue is that not charging for the control plane and charging for the control plane leads to two very different Kubernetes architectures, and as per your docs, those decisions made at t…

I agree the rollout is a little bumpy but I'm curious what workloads you are using k8s for where a $74/mo (or $300/mo) bill isn't a rounding error in your capex?

It's charged per-cluster. GKE encouraged (and was great for) running multiple clusters for all kinds of isolation and security reasons.

This cost increases rapidly for those scenarios.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#458

Post by theptip » Thu, Mar 05, 2020, 1:20 AM UTC

bcheung wrote: post #364

pojzon wrote: post #107

Earlier quoted context omitted.

How about managing own k8s running on VMs / bare-metal? Pretty much anyone who works in ops longer understood from the go that its impossible to be totally provider-agnostic. K8S is just a nice api on top of provider api that still requires provider specific configuration.

I'm running my own on bare metal dedicated servers. You will need to install a few extra things (MetalLB for LoadBalancer, CertManager for SSL, an ingress controller (nginx, Ambassador, Gloo), and one of the CSI plugins for your preferred storage method). It is extra work but as a personal cluster for hobby work, I'm paying $65/mo total for the cluster. Same specs would probably be $1000/mo at a public cloud provider…

MetalLB looks fun, hadn't seen that one.

If you want something production-grade (i.e. doesn't say "beta" on the tin) then I think Calico should solve most of the same problems too (it does BGP peering to your ToR switch):

https://docs.projectcalico.org/networking/determine-best-net...

Does MetalLB do something extra I'm missing?

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#459

Post by freedomben » Thu, Mar 05, 2020, 1:21 AM UTC

JeremyNT wrote: post #450

freedomben wrote: post #168

Earlier quoted context omitted.

Disclaimer: I work for Red Hat and am very biased, but this is my own honest opinion. If you're going to run on bare-metal or in your own VMs, OpenShift is very much worth a look. There are hundreds, maybe thousands of ways to shoot yourself in the foot, and OpenShift puts up guard rails for you (which you can bypass if you want to). OpenShift 4 runs on top of RHCOS which makes node management much simpler, and allow…

So you move from vendor lock in with the cloud provider, to vendor lock in with an expensive, proprietary* IBM k8s distribution with its strange nonstandard opinions about workflows that you have to manage yourself? Don't get me wrong, I appreciate RHAT's code contributions very much, they have done a lot for k8s! But running OKD on one's own is a bad idea, while paying for IBM support makes you as much a hostage as…

Thanks for the edits (and acknowledging our contributions). I wasn't sure if you were just trolling or not before, so I didn't want to engage.

Your concern is valid, and I agree with you that OKD is not supported enough. I have my own theories as to why, but I will keep my criticism "in the family" (but do know there are people that want to see OKD be a first-class citizen, and know we are falling short right now). We had some challenges supporting OKD 4.x because the masters in 4.x now require Red Hat CoreOS (and nodes it is highly recommended), but RHCOS was not yet freely available. This is obvoiusly a big problem. Now that Fedora CoreOS is out, there is a freely distributable host OS on which to build OKD, so it will be better supported and usable. FWIW I have a personal goal to have a production-ish OKD cluster running for myself by end of the year.

I'll admit I am a little offended at being called a "proprietary IBM K8s distribution," but I don't think you meant to be offensive. IBM has nothing to do with OpenShift, beyond the fact that they are becoming customers of it. Every bit of our OpenShift code is open source and available. You are right that it's not in a production-usable state (although there are people using it) but it's a lot better than you'll get from other vendors. We are at least trying to get it to a usable state, unlike many of them. We are strapped for resources like everyone else, and K8s runs a mile a minute and requires significant effort to stay ahead). This space is still really young and really hot, and I am confident we'll get the open source in a good, usable, state, much like Fedora and CentOS are. I also don't think OpenShift is really all that expensive considering the enormous value it provides. The value really does shine at scale, and may not be there for smaller shops.

I don't blame you for waiting, I probably would too. Our current offering is made for enterprise scale, so isn't tailored to everyone. I've heard OpenShift online has gotten better, but haven't tried it myself. Eventually I plan to run all my personal apps on OKD (I have maybe a dozen, mostly with the only users being me and my family), but until then I've been using podman pods with systemd, which will be trivial to port to OKD once it's in a good state.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#460

Post by jrockway » Thu, Mar 05, 2020, 1:22 AM UTC

busterarm wrote: post #444

derefr wrote: post #423

Earlier quoted context omitted.

Why would one want lots of little GKE clusters, anyway? Google itself doesn't part up its clusters this way, AFAIK. I don't want a cluster of underutilized instances per application tier per project; I want a private Borg to run my instances on—a way to achieve the economies-of-scale of pod packing, with merely OS-policy-level isolation between my workloads, because they're all my workloads anyway. (Or, really, I'd r…

We run lots of small clusters in our projects and identical infrastructure/projects for each of our environments. Multiple clusters lets us easily firewall off communication to compute instances running in our account based on the allocated IP ranges for our various clusters (all our traffic is default-deny and has to be whitelisted). Multiple clusters lets us have a separate cluster for untrusted workloads that have…

Namespaces handle most of these issues. A NetworkPolicy can prevent pods within a namespace from initiating or receiving connections from other namespaces, forcing all traffic through an egress gateway (which can have a well-known IP address, but you probably want mTLS which the ingress gateway on the other side can validate; Istio automates this and I believe comes set up for free in GKE.) Namespaces also isolate pods from the control plane; just run the pod with a service account that is missing the permissions that worry you, or prevent communication with the API server.

GKE has the ability to run pods with gVisor, which prevents the pod from communicating with the host kernel, even maliciously. (I think they call these sandboxes.)

The only reason to use multiple clusters is if you want CPU isolation without the drawbacks of cgroups limits (i.e., awful 99%-ile latency when an app is being throttled), or you suspect bugs in the Linux kernel, gVisor, or the CNI. (Remember that you're in the cloud, and someone can easily have a hypervisor 0-day, and then you have no isolation from untrusted workloads.)

Cluster-scoped (non-namespaced) resources are also a problem, though not too prevalent.

Overall, the biggest problem I see with using multiple clusters is that you end up wasting a lot of resources because you can't pack pods as efficiently.