Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Kubernetes Engine is introducing a cluster management fee on June 6

cloud.google.com

Post reply on HN
Print view View original

611–620 of 633 posts

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#611

Post by snisarenko »

darau1 wrote: post #149
rolleiflex wrote: post #58

This is awful - I don’t think GCP is fully aware of their position in the market as the second, inferior choice. I took a bet on the underdog by using GCP and they bit me back in return. Especially considering their ‘default’ kubernetes config automatically sets you up with three(!) control planes in replication, that’s, as far as I understand, $~300 added to our monthly bill, for nothing. Oh, and per their docs, thr…

> their position in the market as the second, inferior choice Who's the superior choice? EKS?

I've been using digital ocean k8 cluster. It's been pretty good. (a few kinks with configuring load balancers, but other than that its good).

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#612

Post by freedomben »

shaklee3 wrote: post #605
freedomben wrote: post #598

Earlier quoted context omitted.

There are several reasons, many of them are that the nodes themselves are managed by OpenShift operators. If you run (as cluster admin) `oc get clusteroperators` you'll see plenty that are for hosts, such as an upgrader and a tuner. If the operators had to be distro agnostic it would be a support nightmare, and we wouldn't be able to do it. With RHCOS (immutable OS) we also have enough guarantees to safely upgrade sy…

Can you describe what kind of tuning the rhel tuning tools do that are not available using the normal kernel constructs? Last I checked tuna and others did everything you could do in Ubuntu, but without knowing the guts of the system. Again, I think the idea of OS is great, but you've lost us, and likely other big customers because of that restriction. Having old kernels is just not an option for some people.

I'm not informed enough to tell you what the tuning tools do, so I'll dodge that question. But "[h]aving old kernels is just not an option for some people" is exactly the type of problem this solves. You literally don't have to know or care what kernel your node runs, because it doesn't matter! The OS is a very thin layer underneath K8s, a layer which is entirely managed by applications running as pods (supervised by an operator) on the system. Whatever apps/daemons/services you need to run move to pods on OpenShift. If you need to manage the node itself there is an API for it. If you truly need underlying access, then this is not for you, but you'd be amazed at how many people (myself included) started out balking at this and thought "no way, for compliance we need " but after re-thinking the system realized you really don't. By "complicating" the system with immutable layers, we actually simplify the system. It was much like learning functional programming to me. By "complicating" programming by taking away stuff (like global variables, side-effects, etc) it actually simplified it and reduced bugs by a huge margin.

If you are like me and are old school and think "huh, yeah that makes me nervous" I completely understand that, but we've seen some serious success with it. I'm a skeptical person, and telling me I can't SSH to my node freaks me out a bit, but I'm becoming a convert.

I would also note if you buy OpenShift you get the infrastructure nodes (masters, and some workers for running openshift operator pods) for free (typically, but I'm not a salesperson so don't hold me to that if I've misspoke :-P), so you aren't paying for the super locked in OS. I suppose you do have to pay for RHEL8 or RHCOS on the worker nodes running your pods, and we don't support other distros (because we expect a very specific selinux config, CRI-O config (container runtime), among other things), so I guess there's some dependence there, although I recommend RHCOS for all your nodes and then just use the Machine API if you need it.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#613

Post by snisarenko »

trilinearnz wrote: post #309
rolleiflex wrote: post #58

This is awful - I don’t think GCP is fully aware of their position in the market as the second, inferior choice. I took a bet on the underdog by using GCP and they bit me back in return. Especially considering their ‘default’ kubernetes config automatically sets you up with three(!) control planes in replication, that’s, as far as I understand, $~300 added to our monthly bill, for nothing. Oh, and per their docs, thr…

Who would be the first choice: Amazon or Microsoft?

Give digital ocean k8 cluster a try. It's been pretty decent for me

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#614

Post by zekrioca »

patrec wrote: post #609
zekrioca wrote: post #604

Earlier quoted context omitted.

In this case, one needs fault-tolerance. One way to achieve it is through replication, where an extra copy (or perhaps a reconstruction receipt) of your DB instance runs somewhere else. Usually DBs achieves this through transactions. Additionally, you can have distributed DBs, which then use distributed transactions for achieving so. I am not expert, but K8s handles task replication, and either spawn or route a reque…

Distributed transactions are a non-starter – I already said I want to run master-slave with synchronous replication, which is basically what you want to do in >99% of cases where you have a DB with important stuff in it.

It is not really about what you want, but about how to migrate the DB* to another location while also minimizing its downtime/slower-ness.

You need to instantiate a secondary DB replica somewhere else and start the DB migration. Since there will be "two instances" of the same DB running, you will also need to set up a (temporary) proxy for routing and handling the DB requests w/ something like this:

1) if the data being requested is already migrated, request is handled by the (new) secondary replica. 2) Primary instance handles the request, otherwise 2.1) Requested data should be migrated to secondary replica (asynchronously, but note that a repeated request may invalidate a migration).

Turn the proxy router off once the whole state of your primary DB instance is fully migrated, making the secondary replica the primary one. That's really just a napkin recipe for completing a live migration, though.

* We are now getting into the distributed transaction world because you can never be 100% sure that writing to 2 databases can succeed or fail at the same time. There is this talk from this guy who deals with similar problem you have: http://www.aviransplace.com/2015/12/15/safe-database-migrati...

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#615

Post by aaronblohowiak »

yoshiat wrote: post #597
aaronblohowiak wrote: post #590

Earlier quoted context omitted.

I’m sorry if i am reading it wrong, but this guide to multi-tenancy seems to suggest not being multi-tennant and instead running a separate cluster per project. This seems more like scaling single-tenancy than multi-tenant (no bin packing oppty for instance.) or did i read it wrong?

Sorry if it was confusing. You need to read into more about how to set up in-cluster multi-tenancy. We do recommend robust configurations for production setup (e.g. dev, staging and production) however you can certainly squash and skip it if not necessary. Thanks for the feedback though. We'll consider adding such notes explicitly.

>You need to read into more about how to set up in-cluster multi-tenancy.

I am trying to do that. Where would you suggest? Throughout the comments on this post, when people suggest namespace, pod or node level separation you ask them to PTAL and read the link which suggests the single-tenant cluster-per-project approach (that is under the Multi-tenant cluster, confusingly.) The link you sent talks about cluster-per-project, which is not multi-tenancy as I understand it. Perhaps a different name would be less confusing (robust federated cluster administration?)

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#616

Post by jiggawatts »

varshithr wrote: post #554
jiggawatts wrote: post #550

Earlier quoted context omitted.

Cloud providers assume everyone is just like them, or like Netflix: Load balancers in clusters balancing groups of clusters. Clusters of clusters. Many regions. Many availability zones. Anything can be lost, because an individual data centre is just 5% of the total, right? Meanwhile most of my large government customers have a couple of tiny VMs for every website. That's it. That's already massive overkill because th…

Have you considered the Google App Engine on GCP in standard mode? That seems like a good fit based on your explanation but I could be wrong.

I had a similar conversation with a government customer, saying that they should pool their web applications into a single shared Azure Web App Service Plan, because then instead of a bunch of small "basic" plans they could get a "premium" plan and save money.

They rejected it because it's "too complex to do internal chargebacks" in a shared cluster model.

This is what I mean: The cloud is for orgs with one main application, like Netflix. It's not ready for enterprises where the biggest concern is internal bureaucracy.

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#617

Post by yoshiat »

aaronblohowiak wrote: post #615
yoshiat wrote: post #597

Earlier quoted context omitted.

Sorry if it was confusing. You need to read into more about how to set up in-cluster multi-tenancy. We do recommend robust configurations for production setup (e.g. dev, staging and production) however you can certainly squash and skip it if not necessary. Thanks for the feedback though. We'll consider adding such notes explicitly.

>You need to read into more about how to set up in-cluster multi-tenancy. I am trying to do that. Where would you suggest? Throughout the comments on this post, when people suggest namespace, pod or node level separation you ask them to PTAL and read the link which suggests the single-tenant cluster-per-project approach (that is under the Multi-tenant cluster, confusingly.) The link you sent talks about cluster-per-p…

"This guide provides best practices to safely and efficiently set up multiple multi-tenant clusters for an enterprise organization."

This "multiple" multi-tenant clusters part isn't coming through. Please do jump into "Securing the cluster" section to cut corners and learn what to do in a single cluster. We're fixing the sections to avoid the confusions. Thanks for the feedback!

https://cloud.google.com/kubernetes-engine/docs/best-practic...

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#618

Post by ickyforce »

Glyptodon wrote: post #540
ones_and_zeros wrote: post #175

Earlier quoted context omitted.

I agree the rollout is a little bumpy but I'm curious what workloads you are using k8s for where a $74/mo (or $300/mo) bill isn't a rounding error in your capex?

My boss viewed it as the main way to deploy containerized systems offered by cloud providers and figured we could run most of our internal only things in it for a couple hundred a month - we don't really need the guarantees and scale, and he saw it as a way to avoid creating excess numbers of dedicated VMs, as cloud run isn't sufficient for our non-static stuff. This view up until now has actually been quite accurate…

E.g. https://aws.amazon.com/ecs/ - that was quite nice https://aws.amazon.com/fargate/ - haven't tried

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#619

Post by w9jds »

There are so many issues with this. I used GCP because I liked the service and was able to build all of my stuff well on it. My personal projects already make me shell out over $250 a month because of running costs and now you want to gouge me for hourly running of a cluster I alraid pay hourly of compute time for? What in the actual hell...

Re: Google Kubernetes Engine is introducing a cluster management fee on June 6

#620

Post by developer2 »

sho wrote: post #506
developer2 wrote: post #484

Earlier quoted context omitted.

Care with DigitalOcean's Kubernetes offering. Unless they have completely revamped their entire stack since their initial release, it's a security nightmare. The way they launched it was unacceptable for any company to use. I immediately migrated my projects away and closed my account upon understanding how abysmally they screwed up the security of that offering. The fact they were willing to launch that means they c…

Are you referring to this? https://www.4armed.com/blog/hacking-digitalocean-kubernetes/ I'm curious as to whether that has been fixed and a proper security evaluation done. I've been avoiding k8s on DO for a while because of that (perfectly comfortable with their other services) but it would be good to get an update.

Yeah, that's the one. I'd done my own analysis before that came out and figured it was an unacceptable risk, and then that article came out with the actual attack vector in all its glory. The simple fact the k8s/etcd ports are exposed on a public address with no ability to firewall it off is bad enough, as you're relying on the security of the software running on those ports rather than a firewall restricting which source address(es) can even connect to begin with.

The credentials (certificates) being exposed via http://169.254.169.254/metadata/v1/user-data – from within any pod/container, not just from a physical node – was the final straw. I'd forgotten that the DO token wasn't directly listed there, but can be extracted from the etcd instance where it is stored (explained under "DigitalOcean Account Takeover" of that article).

Again, all of this may have been (and hopefully has been) mitigated since the original release. For me, it's too late to reevaluate; the fact that was considered releasable in the first place destroyed any credibility in my eyes.

Post reply on HN