Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes 1.3 released

github.com

Post reply on HN
Print view View original

51–60 of 94 posts

Re: Kubernetes 1.3 released

#51

Post by justinsb »

Rapzid wrote: post #43

I'm still sitting on the sidelines waiting for the easy to install, better documented for AWS version. It's also a bit unclear as to why we are talking federation and master/slave in 2016; other systems are using raft and gossip protocols to build masterless management clusters.. Watching issues like https://github.com/kubernetes/kubernetes/issues/23478 , and https://github.com/kubernetes/kubernetes/issues/23174 .. I…

I do a lot of work for k8s on AWS (e.g. I'm to blame for the original under-the-hood document). I agree with most of what you've said we need to improve on, and we are working on it! We added a bunch of ELB configuration options in 1.3 and we'll likely complete the ELB feature set in 1.4 (it probably would have landed in 1.4 but for my tardiness in doing reviews). In addition, there's a well-known "trick" where you set up a service using a NodePort and then set up the ELB to point to that, if you want to go beyond what k8s offers (e.g. if you want to reuse an existing ELB etc). What our docs are lacking, we make up for with an excellent community (for AWS discussions #sig-aws on the k8s slack).

The debate between automation vs simplification is one that has gone on since k8s 1.0 and likely will continue to be had. But I think to an extent it is a false choice: I created a new k8s installation/ops tool (i.e. did work on the "automation" side), and out of that I got a fairly clear road-map for how we could simplify the installation dramatically in 1.4. In other words, some of the simplification you ask for comes by embedding the right pieces of the automation. k8s is open-source, so I have to persuade others of this approach, but I think that's a good thing, and I'd hope you'd join in that discussion also (e.g. #sig-cluster-lifecycle on the k8s slack).

Re: Kubernetes 1.3 released

#52

Post by Rapzid »

lobster_johnson wrote: post #47
Rapzid wrote: post #44

Earlier quoted context omitted.

I can't agree with this enough. We are all on AWS and the level of effort it would take to migrate to Kubernetes while maintaining our ability to spin up complete ad-hoc environments on the fly(which also serves as continual DR testing) seems too much to justify at this point. Also, I can't come out the other side with just one or two people understanding, or having any hope of understanding, how everything works :|…

One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work. All you need to do, in broad strokes, is: * Set up a VPC. Defaults work. * Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs. * Install Kubernetes…

Awesome, I'll take a look into all that! This doesn't look too bad. Do you know if you can combine the masters/minions? Our environments are VPC isolated, and we support ad-hoc creation so I'd like to keep server count requirements to a bare minimum.. The current from-scratch guide says it is not necessary to make the distinction between master nodes and normal nodes; and the api, controller, etc appear to be hosted as pods. This makes me happy and makes sense, but then you have something like this which has me confused: https://github.com/kubernetes/kubernetes/issues/23174 .

On a side note, it's pretty awesome how Docker embedded the key-value store into the main binary. Appears to reduce complexity quite a bit.

Re: Kubernetes 1.3 released

#53

Post by lobster_johnson »

Rapzid wrote: post #52
lobster_johnson wrote: post #47

Earlier quoted context omitted.

One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work. All you need to do, in broad strokes, is: * Set up a VPC. Defaults work. * Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs. * Install Kubernetes…

Awesome, I'll take a look into all that! This doesn't look too bad. Do you know if you can combine the masters/minions? Our environments are VPC isolated, and we support ad-hoc creation so I'd like to keep server count requirements to a bare minimum.. The current from-scratch guide says it is not necessary to make the distinction between master nodes and normal nodes; and the api, controller, etc appear to be hosted…

You can run them on the same box just fine. There's nothing magical about any of those processes.

However, using dedicated masters (by which I mean mostly kube-apiserver) separate from worker nodes is a good idea to avoid high load impacting the API access.

(Just keep in mind that the Kismatic packages I referred to won't support this — you can't install kubernetes-master and kubernetes-node at the same time. But as you discovered, you can run everything except kubelet as pods. On the other hand, kube-apiserver needs a whole bunch of mounts as well as host networking, so to me it seems like you don't gain all that much.)

What is this Docker key-value store you mention?

Re: Kubernetes 1.3 released

#54

Post by justinsb »

Rapzid wrote: post #49
thockingoog wrote: post #45

Earlier quoted context omitted.

It's funny how words can be played. The Kubernetes "master" is a set of 1 or more machines that run the API server and associated control logic. This is exactly what systems like Docker swarm do, but they wrap it in terms like RAFT and gossip that make people weak in the knees. Kubernetes has RAFT in the form of the storage API (etcd). This is a model that has PROVEN to work well, and to scale well beyond what almost…

I would consider "kicking the tires" actually running up a cluster and playing with it. One can also evaluate by reading documentation and others reports of issues to look for show-stopping problems. For instance, a couple releases ago there was not multi-AZ support. The word on the street at that time was to create multiple clusters and do higher level orchestration across them.. That was a no-go for us; no need to…

This frustrates me greatly, only because I agree with you so vehemently :-) We have an open issue tracker where real decisions are made, and so engineers will argue about different approaches. Compare to alternatives, where you see demos that are double-acts of good-cop vs good-cop where apparently there are no trade-offs and everything is perfect. It isn't my experience that products where the debates are hidden are better; it is certainly easier to see the compromises when the debates are public.

So: there was a big discussion about whether a single k8s cluster should span multiple AZs (which shipped in 1.2), or whether we should allow the API to target multiple independent clusters (federation, the first version of which is shipping in 1.3). The core of the argument is that multi-zone is simpler for most users, but with only one control plane it is less reliable than a federation of totally independent clusters. Federation also brings other benefits, like solving the problem of running in clusters that are not in a single "datacenter" i.e. where you need to worry about non-uniform latency. I haven't seen anyone else make a serious attempt at solving this.

So, remember that the issue tracker is filled with the unvarnished discussions that come from true open source development. I think it is an asset for you, because you don't discover those things 3 months into using your chosen product; but it is definitely a liability for k8s, because we rely on you realizing this in your initial evaluation and weighting appropriately (the devil you know vs the devil you don't). I think k8s is likely much better than you think it is, and you should come talk to us on slack and make sure of that fact! It certainly sounds like you have an interesting use case that we'd like to hear about and consider.

But yes, our docs should be better!

Re: Kubernetes 1.3 released

#55

Post by Rapzid »

lobster_johnson wrote: post #53
Rapzid wrote: post #52

Earlier quoted context omitted.

Awesome, I'll take a look into all that! This doesn't look too bad. Do you know if you can combine the masters/minions? Our environments are VPC isolated, and we support ad-hoc creation so I'd like to keep server count requirements to a bare minimum.. The current from-scratch guide says it is not necessary to make the distinction between master nodes and normal nodes; and the api, controller, etc appear to be hosted…

You can run them on the same box just fine. There's nothing magical about any of those processes. However, using dedicated masters (by which I mean mostly kube-apiserver) separate from worker nodes is a good idea to avoid high load impacting the API access. (Just keep in mind that the Kismatic packages I referred to won't support this — you can't install kubernetes-master and kubernetes-node at the same time. But as…

https://blog.docker.com/2016/06/docker-1-12-built-in-orchest...

They are using a Raft based store inside the engine now so there is no external etcd dependency. IIRC they are using etcd's raft implementation.

Re: Kubernetes 1.3 released

#56

Post by lobster_johnson »

Rapzid wrote: post #55
lobster_johnson wrote: post #53

Earlier quoted context omitted.

You can run them on the same box just fine. There's nothing magical about any of those processes. However, using dedicated masters (by which I mean mostly kube-apiserver) separate from worker nodes is a good idea to avoid high load impacting the API access. (Just keep in mind that the Kismatic packages I referred to won't support this — you can't install kubernetes-master and kubernetes-node at the same time. But as…

https://blog.docker.com/2016/06/docker-1-12-built-in-orchest... They are using a Raft based store inside the engine now so there is no external etcd dependency. IIRC they are using etcd's raft implementation.

Interesting, thanks. Personally, Docker is already too monolithic, and this just looks like it adds unnecessary coupling to something should be less coupled in the first place. I'd prefer to use etcd.

I think rkt is making some good decisions and is worth keeping an eye on. Not sure I love the tight coupling to systemd, but the fact that it avoids the PID 0 problem and lets containers be their own processes (separate from the "engine", which can choreograph containers through the systemd API, building on all of its process handling logic) are improvements over Docker. In fact, rkt uses the same networking model as Kubernetes.

Re: Kubernetes 1.3 released

#57

Post by bogomipz »

thockingoog wrote: post #40
bogomipz wrote: post #36

Does anybody know how 1.3 is for stateless services? Can I use an API to crete a persistent disk volume and adjust that volume size just as I would any other resource like CPU, memory? The use case being postgres/mysql instances.

You can't, in general, adjust the size of block devices purely transparently. We don't currently support blockdev resizing. Would love to talk about how to achieve that, though

Is it possible under specific circumstances such as if using LVM devices?

Re: Kubernetes 1.3 released

#58

Post by thockingoog »

bogomipz wrote: post #57
thockingoog wrote: post #40

Earlier quoted context omitted.

You can't, in general, adjust the size of block devices purely transparently. We don't currently support blockdev resizing. Would love to talk about how to achieve that, though

Is it possible under specific circumstances such as if using LVM devices?

an LVM device is local, which is not supported as PersistentVolume. :(

Re: Kubernetes 1.3 released

#59

Post by chrissnell »

aedocw wrote: post #35
chrissnell wrote: post #20

Earlier quoted context omitted.

We've seen similar savings at our company. We have deployed Kube on a 6-node cluster of CoreOS nodes with 512GB each. These are dedicated servers hosted at Rackspace. We're about 30-40% utilized on RAM and maybe 15-20% on CPU. To host a similar set of services on our older Openstack environment would require at least 2-3x the number of servers. The cost savings isn't even the best part. Kubernetes has allowed us to b…

Have you compared the cost of maintaining your own CoreOS infrastructure at RAX just for kubernetes to using Googles Container Engine? If your services are all containerized and deployed via k8s to begin with, seems like you wouldn't have much reason to maintain your own infra at that point.

Yes, absolutely. We ran a four month experiment on GCE--we built an off-site logging cluster fluentd+Elasticsearch+Kibana. The performance was decent but the cost of RAM and disk are way higher.

I will tell you that the economics are most definitely not there. This is a common misconception amongst the HN crowd in general--that public cloud infra is cheaper. For small footprints, public cloud makes sense but once you get into the larger footprints (300+ instances), it's far cheaper to lease dedicated hardware or DIY in colocation. We're running on approximately 40 dedicated rackmount servers for Openstack and 6 for Kubernetes. To get the equivalent amount of disk and RAM, we would pay 2-3x at AWS or GCE. We could probably cut our cost by an additional 30% by moving what we have to colo but we would lose some flexibility and would have to take on additional headcount.

From a maintainability standpoint, GCE makes Kubernetes easy which is a good thing if you've never run it before. It's not that hard to run it yourself, though. A senior-level systems engineer will be a Kube master after about two months of use. Just guessing, I think it takes about 1/4 of an engineer-week to support our Kube cluster for a week. I think we could grow our cluster 20x without a significant workload increase for our ops team.

We are in the process of automating the last few manual aspects of our Kubernetes infra: load balancing and monitoring. We're building these in the style that we've built the rest of our pipeline: YAML files in a project repo. Simply drop your Datadog-style monitoring/metrics config and your load balancer spec in your project's Github repo and the deployment pipeline will build out your monitoring, metrics, and LB automatically for you.

Re: Kubernetes 1.3 released

#60

Post by chrissnell »

brokebook wrote: post #41
chrissnell wrote: post #20

Earlier quoted context omitted.

We've seen similar savings at our company. We have deployed Kube on a 6-node cluster of CoreOS nodes with 512GB each. These are dedicated servers hosted at Rackspace. We're about 30-40% utilized on RAM and maybe 15-20% on CPU. To host a similar set of services on our older Openstack environment would require at least 2-3x the number of servers. The cost savings isn't even the best part. Kubernetes has allowed us to b…

so true, the self service aspect is indeed amazing. once developers or qa people grok the concepts and api it can do wonders to your productivity. also, working with k8s will probably spoil you, it's pretty annoying to "go back" to other environments, where you're confronted with problems which would be effortlessly solvable in kubernetes.

It completely spoils you. My team got to do an Openstack cluster migration this month and there was definitely some grousing. The workflow of traditional private clouds is just so tedious and flaky. We could grow our Kubernetes cluster 10x without hiring any additional engineers for the ops team.
Post reply on HN