Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine is introducing a cluster management fee on June 6

cloud.google.com

Post reply on HN
Print view View original

591–600 of 633 posts

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#591

Post by aludwin »

lvh wrote: post #579
aludwin wrote: post #333

Earlier quoted context omitted.

I know kubeflow can use namespaces for its own purposes, but otherwise I thought that was quite rare. Namespaces are intended to be used for exactly this usecase (isolating teams and/or workloads). What kind of system have you seen where this isn't true?

We've seen plenty of examples where people do this. Sometimes it's different teams (e.g. the ML team is namespaced away from the primary customer flow) and sometimes it's for different customers. Really, it sounds like we're in agreement about why this happens, the confusion is just whether or not that normally happens within a company in the course of doing business?

Gotcha. If you're looking for subnamespacing, HNC offers self-service subnamespace creation, have you looked into that? https://github.com/kubernetes-sigs/multi-tenancy/tree/master...

(My apologies if we've chatted about this before in another venue, I'm losing track of whom I've already talked to)

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#592

Post by vasco »

rolleiflex wrote: post #449
carterehsmith wrote: post #350

Earlier quoted context omitted.

I see. Curious about the latency between your GCP apps and the database on AWS - is it like 1 ms or 100ms? Does it affect the product?

About 4ms for us. However, we chose our data centres on both ends very carefully. There are tables online you can find that for those pings, one such is here: https://medium.com/@sachinkagarwal/public-cloud-inter-region... However this means we are paying for egress on both sides. This was something we chose to eat due to GCP Kubernetes, but considering today’s changes, it probably no longer makes sense.

So you decide to eat egress costs in perpetuity, which will scale as you go, but a one time increase of $70 per month is enough to make you go back? What are you even trying to optimize for?

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#593

Post by rcarmo »

dcolkitt wrote: post #266
sethvargo wrote: post #80

Earlier quoted context omitted.

Thank you for the feedback. The management fee is per cluster. You are not billed for replicated control planes. You can use the pricing calculator at https://cloud.google.com/products/calculator#tab=container to model pricing, but it should work out to $73/mo regardless of nodes or cluster size (again, because it's charged per-cluster). There's also one completely free zonal cluster for hobbyist projects.

Hi Seth, What about clusters that are used for lumpy work loads? Like data science pipelines? For example, our org has a few dozen clusters being used like that. Each pipeline gets its own cluster instance as a way to enforce rough and ready isolation. Most of the times the clusters sit unused. To keep them alive we keep a small, cheap, preemptive node alive on the idle cluster. When a new batch of data comes in, we…

Here you go:

https://github.com/rcarmo/azure-k3s-cluster (this is an Azure template that I use precisely for testing that kind of workloads - spinning up one of these, master included, takes a couple of minutes at most).

(full disclosure: I work at Microsoft - Azure Kubernetes Service works fine, but I built the above because I wanted full control over scaling and a very very simple shared filesystem)

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#594

Post by rcarmo »

lazyier wrote: post #211
lazyier wrote: post #148

Earlier quoted context omitted.

Kubespray works well for me for setting up a bare bones kubernetes cluster for the lab. I'll use helm to install metallb for the load balancer, which you can then tie into whatever egress controller you like to use. For persistent storage a simple NFS server is the bees knees. Works very well and a NFS provisioned is a helm install. Very nice, especially, over 10GbE. Do NOT dismiss NFSv4. It's actually very nice for…

> Have you used NFS for persistent storage in prod much? I think NFS is heavily underrated. It's a good match for things like hosting VM images on a cluster and for Kubernetes. In the past I really wanted to use things iSCSI for hosting VM images and such things, but I've found that NFS is actually a lot faster for a lot of things. There are complications to NFS, of course, but they haven't caused me problems. I woul…

I use an SMB file share for my node pools - here's how to set up a non-managed cluster on Azure that does that: http://github.com/rcarmo/azure-k3s-cluster

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#595

Post by rcarmo »

atombender wrote: post #420
blazespin wrote: post #399

Earlier quoted context omitted.

Or move to minikube and friends. If it's a dev environment you can usually get away with such things.

Kubernetes consumes a lot of CPU even when idle, due to the polling design, which makes Minikube is a really poor fit for developer machines. It's well known [1] to sit there eating 20-30% CPU, draining your battery and frustrating your life while doing absolutely nothing. This applies to all the Kubernetes distributions, including Kind and Docker Desktop. Not sure if the same applies to K3s, though. [1] https://gith…

It does. My k3s setups (like https://github.com/rcarmo/azure-k3s-cluster) take up nearly 100% of the puny master node I allocate to them, and kill my Raspberry Pi SD cards as well.

Swarm, in comparison, is much friendlier (and you can use it for dev/test across multiple machines just fine)

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#596

Post by rcarmo »

JeremyNT wrote: post #450
freedomben wrote: post #168

Earlier quoted context omitted.

Disclaimer: I work for Red Hat and am very biased, but this is my own honest opinion. If you're going to run on bare-metal or in your own VMs, OpenShift is very much worth a look. There are hundreds, maybe thousands of ways to shoot yourself in the foot, and OpenShift puts up guard rails for you (which you can bypass if you want to). OpenShift 4 runs on top of RHCOS which makes node management much simpler, and allow…

So you move from vendor lock in with the cloud provider, to vendor lock in with an expensive, proprietary* IBM k8s distribution with its strange nonstandard opinions about workflows that you have to manage yourself? Don't get me wrong, I appreciate RHAT's code contributions very much, they have done a lot for k8s! But running OKD on one's own is a bad idea, while paying for IBM support makes you as much a hostage as…

I work for Microsoft and I quite like OpenShift (although we have AKS as the default, "vanilla" managed k8s service, you can also run OpenShift on Azure).

Sure it's opinionated, but at this point, which flavor of k8s isn't? Even k3s (which I play around with on https://github.com/rcarmo/azure-k3s-cluster) has its quirks.

Everyone who's targeting the enterprise market has a "value-added" brew of k8s at this point, so kudos to RH for the engineering effort :)

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#597

Post by yoshiat »

aaronblohowiak wrote: post #590
yoshiat wrote: post #368

Earlier quoted context omitted.

Yes, namespace alone isn't sufficient for isolation. Would you be able to look at our latest Multi-Tenancy best practices? https://cloud.google.com/kubernetes-engine/docs/best-practic... It's a living product which comes with Terraform modules. We introduced various features to enable doing Multi-Tenancy as well (and more on their way!)

I’m sorry if i am reading it wrong, but this guide to multi-tenancy seems to suggest not being multi-tennant and instead running a separate cluster per project. This seems more like scaling single-tenancy than multi-tenant (no bin packing oppty for instance.) or did i read it wrong?

Sorry if it was confusing. You need to read into more about how to set up in-cluster multi-tenancy.

We do recommend robust configurations for production setup (e.g. dev, staging and production) however you can certainly squash and skip it if not necessary.

Thanks for the feedback though. We'll consider adding such notes explicitly.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#598

Post by freedomben »

shaklee3 wrote: post #516
freedomben wrote: post #459

Earlier quoted context omitted.

Thanks for the edits (and acknowledging our contributions). I wasn't sure if you were just trolling or not before, so I didn't want to engage. Your concern is valid, and I agree with you that OKD is not supported enough. I have my own theories as to why, but I will keep my criticism "in the family" (but do know there are people that want to see OKD be a first-class citizen, and know we are falling short right now). W…

Honestly, the fact that it requires rhel or centos was enough to make it not feasible for us. I wish that would change, since I can't think of any reason the distribution should affect openshift.

There are several reasons, many of them are that the nodes themselves are managed by OpenShift operators. If you run (as cluster admin) `oc get clusteroperators` you'll see plenty that are for hosts, such as an upgrader and a tuner. If the operators had to be distro agnostic it would be a support nightmare, and we wouldn't be able to do it. With RHCOS (immutable OS) we also have enough guarantees to safely upgrade systems without human intervention. Can you imagine doing that in an Enterprise environment while trying to support multiple distributions? I can't.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#599

Post by Kassius509 »

rolleiflex wrote: post #58

This is awful - I don’t think GCP is fully aware of their position in the market as the second, inferior choice. I took a bet on the underdog by using GCP and they bit me back in return. Especially considering their ‘default’ kubernetes config automatically sets you up with three(!) control planes in replication, that’s, as far as I understand, $~300 added to our monthly bill, for nothing. Oh, and per their docs, thr…

Come on over to AWS. https://aws.amazon.com/activate/

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#600

Post by freedomben »

merb wrote: post #538
freedomben wrote: post #459

Earlier quoted context omitted.

Thanks for the edits (and acknowledging our contributions). I wasn't sure if you were just trolling or not before, so I didn't want to engage. Your concern is valid, and I agree with you that OKD is not supported enough. I have my own theories as to why, but I will keep my criticism "in the family" (but do know there are people that want to see OKD be a first-class citizen, and know we are falling short right now). W…

btw. I would never run OpenShift after the CoreOS debacle. This was a really sketchy move and still is. Yes RH did a lot for k8s, but killing of a working distribution without a direct migration path that is like "start again". will make your customers angry. also I think the OpenShift terminology is way too much and OpenShift should be a way more thinner layer on top of k8s.

I agree, the CoreOS thing went down grossly. It was a technical nightmare tho. They deeply merged CoreOS and Fedora/RHEL and created a hybrid animal. Creating an upgrade path would have been an insane challenge, and in the end the advice would have been to rebuild anyway to avoid unforeseen issues. They could have left CoreOS repos and stuff up tho and given a longer transition period.
Post reply on HN