Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container orchestration: Moving from fleet to Kubernetes

coreos.com

Post reply on HN
Print view View original

61–70 of 74 posts

Re: Container orchestration: Moving from fleet to Kubernetes

#61

Post by KJKingJ »

Perceptes wrote: post #25
dpratt wrote: post #18

This is interesting, but has a potential problem - what do you use to schedule the control plane? Right now, we use Fleet to schedule a highly available k8s API server and associated singleton daemons. Then API server is required to get anything else scheduled in the cluster. How are they going to solve this bootstrap problem?

As moondev pointed to, eventually bootkube will handle bootstrapping k8s clusters. At my company we just set everything up using cloud-config. A systemd unit boots the kubelet on each server, and static k8s manifests are loaded by the kubelet to run the rest of the k8s components as pods. This way, the kubelet itself is the only component that is not managed by k8s itself.

> At my company we just set everything up using cloud-config. A systemd unit boots the kubelet on each server, and static k8s manifests are loaded by the kubelet to run the rest of the k8s components as pods.

This is the exact same methodology that i've been using and it's worked rather well. The current CoreOS documentation [1] on running Kubernetes follows this methodology too.

[1] https://coreos.com/kubernetes/docs/latest/getting-started.ht...

Re: Container orchestration: Moving from fleet to Kubernetes

#62

Post by dpratt »

Perceptes wrote: post #25
dpratt wrote: post #18

This is interesting, but has a potential problem - what do you use to schedule the control plane? Right now, we use Fleet to schedule a highly available k8s API server and associated singleton daemons. Then API server is required to get anything else scheduled in the cluster. How are they going to solve this bootstrap problem?

As moondev pointed to, eventually bootkube will handle bootstrapping k8s clusters. At my company we just set everything up using cloud-config. A systemd unit boots the kubelet on each server, and static k8s manifests are loaded by the kubelet to run the rest of the k8s components as pods. This way, the kubelet itself is the only component that is not managed by k8s itself.

This is precisely what we do on our cluster as well for the node-level daemons - kubelet and kube-proxy.

We use fleet to schedule the HA API server. You cannot use the Kubelet to schedule this, because you need an API server to schedule cluster-wide pods.

The only solution I can see is to have a config that launches a special 'master' node that runs the API server, but this is uncompelling to me. I'd rather have every single node be identical, and get the API server to pop up somewhere in the cluster using a master election process - which is precisely what fleet does.

Re: Container orchestration: Moving from fleet to Kubernetes

#63

Post by cookiecaper »

raesene6 wrote: post #54

I think Kubernetes is a really interesting product and obviously has a lot of momentum. That said for something thats seeing wide adoption it still has a lot of rough edges and things that need fleshed out. One I ran across recently was the upgrade process for clusters. Per ( https://kubernetes.io/docs/admin/cluster-management/#upgradi... ) it seems that unless you're on GCE the best way to upgrade a cluster is by re…

>That said for something thats seeing wide adoption it still has a lot of rough edges and things that need fleshed out.

Yes, I'm concerned about this not just with k8s, but Docker as well. Both are very immature products and there's a massive rush to adopt them, attributable almost entirely to social pressures and the insecurities of people who lead these tech depts.

When things like StatefulSets and persistent storage are still iffy/under development, it should be clear that these things are nowhere near production-ready.

Re: Container orchestration: Moving from fleet to Kubernetes

#64

Post by merb »

puzzle wrote: post #49
merb wrote: post #48

I don't get that move. fleet was extremly well suited to schedule a kubernetes high available master. as soon as you have 3 etcd nodes and 3 fleet nodes you could use fleet to bootstrap kubernetes in a way more stable fashion than all of the other available options. if people remove the low level tools to manage a cluster it will be harder and harder to bootstrap higher level stuff. but well, what to expect in the co…

You can do that kind of bootstrap without fleet. Just use ignition or cloud-config with the right systemd units and a bunch of fixed IP addresses. I think the CoreOS folks worked on a number of ways to simplify and automate bootstrapping of the Kubernetes control plane, so they saw fleet as redundant now. Besides, it took a long time for it to get something resembling a mechanism that updates units in the cluster. Th…

with fleet you could have a single kubernetes master that would've been started on another node, as soon as one node would go down. that won't work with just systemd units.

Re: Container orchestration: Moving from fleet to Kubernetes

#65

Post by eicnix »

raesene6 wrote: post #57
eicnix wrote: post #55

Earlier quoted context omitted.

Building a cluster from scratch is usually not a bad idea: You create a new cluster with the upgraded version, combine both clusters through federation and start moving pods from the old to the new cluster. Upgrading a cluster in place will come in the future.

Whilst for major upgrades that might make sense, what about instances like a high risk security fix where upgrade speed is important... People don't want to be re-building from scratch in that kind of setup...

I fully understand your issue. Creating a new cluster means for me running a script that sets up a new cluster in ~15min. There is https://github.com/apprenda/kismatic which can help simplify your cluster setup if you run in a enterprise environment.

You can also take a look at https://coreos.com/tectonic where coreos provides a enterprise kubernetes distribution that supports updating a kubernetes cluster without downtime but I personally haven't tested tectonic.

Re: Container orchestration: Moving from fleet to Kubernetes

#66

Post by puzzle »

merb wrote: post #64
puzzle wrote: post #49

Earlier quoted context omitted.

You can do that kind of bootstrap without fleet. Just use ignition or cloud-config with the right systemd units and a bunch of fixed IP addresses. I think the CoreOS folks worked on a number of ways to simplify and automate bootstrapping of the Kubernetes control plane, so they saw fleet as redundant now. Besides, it took a long time for it to get something resembling a mechanism that updates units in the cluster. Th…

with fleet you could have a single kubernetes master that would've been started on another node, as soon as one node would go down. that won't work with just systemd units.

I'm sure there are good use cases for Fleet, but running Kubernetes (or anything else) with just one master is asking for trouble.

Re: Container orchestration: Moving from fleet to Kubernetes

#67

Post by benjamin_mahler »

davidmr wrote: post #28
sytse wrote: post #7

Earlier quoted context omitted.

I too salute CoreOS for doing the right thing for their customers and the ecosystem. Kubernetes was something that was hard to predict, it didn't grow organically but was suddenly released by Google. Right now I believe Kubernetes is the project with the most accepted pull requests per day. This came up in a talk from GitHub at Git Merge 2017. It shows that k8s is on its way to becoming the default container schedule…

Completely agreed. The landscape 2-3 years ago looked incredibly different, and I picked Mesos for a rather ambitious project. After it became relatively clear that k8s was going to eclipse Mesos and not by a little bit, trying to unwind that decision basically cost me my job through some political infighting. It's a shame, but such is the price of early adoption, I guess. Given the same information, I'm really confi…

Hi David, can you reveal some of your environment (e.g. on-prem vs cloud), were there any technical reasons for switching or was it primarily a matter of perception of velocity/popularity between the two projects?

Just to add some of my own perception as someone who works on Mesos, Mesos continues to be popular with large technology companies that don't make their technical investments lightly: Twitter, Apple, Netflix, Uber, Yelp, for example. Companies continue to choose a Mesos stack based on its technical merits. The project is still moving fast and adding powerful primitives to support the needs of production environments while distributions like DC/OS are trying to make Mesos more approachable (easy to install, administer) and comprehensive (providing solutions for load balancing, logging, metrics, etc). I hope you will take another look at the Mesos ecosystem at some point, a lot of care has gone into it :)

Re: Container orchestration: Moving from fleet to Kubernetes

#68

Post by mentat »

johnhenry wrote: post #60
hackerboos wrote: post #21

Earlier quoted context omitted.

brew cask install minikube brew install kubectl Then follow the tutorial https://github.com/kubernetes/minikube#quickstart

Thanks for the quick tip! Might I suggest you change that first line to `brew cask install minikube virtualbox`?

brew install docker-machine-driver-xhyve is a friendly laptop choice.

Re: Container orchestration: Moving from fleet to Kubernetes

#69

Post by mentat »

zebra9978 wrote: post #42
predakanga wrote: post #36

Earlier quoted context omitted.

The issue that I have with the newly imagined Docker Swarm is that it continues Docker's trend of trying to be a jack of all trades. I have no specific examples for Docker Swarm, but using this approach in other areas has led to some pretty major deficiencies in Docker's design that they have been slow to fix, and I'm not keen on seeing that happen again. For a concrete example, see https://github.com/docker/docker/i…

I see. incidentally the embedded DNS feature is fairly extensively leveraged by kubernetes - it takes of the situations where you dont want to muck around with underlying /etc/hosts (on the actual metal) and do your changes only on the containers. But I'm hearing what you are saying more and more - Docker Inc is having a huge PR problem. Docker Swarm may actually be good, but people are generally disliking the organi…

They have a PR problem because they break interfaces in minor releases and have actively pushed back on criticism for doing so as "they need to be agile" and "can't be boxed in by competitors".

Re: Container orchestration: Moving from fleet to Kubernetes

#70

Post by mentat »

benjamin_mahler wrote: post #67
davidmr wrote: post #28

Earlier quoted context omitted.

Completely agreed. The landscape 2-3 years ago looked incredibly different, and I picked Mesos for a rather ambitious project. After it became relatively clear that k8s was going to eclipse Mesos and not by a little bit, trying to unwind that decision basically cost me my job through some political infighting. It's a shame, but such is the price of early adoption, I guess. Given the same information, I'm really confi…

Hi David, can you reveal some of your environment (e.g. on-prem vs cloud), were there any technical reasons for switching or was it primarily a matter of perception of velocity/popularity between the two projects? Just to add some of my own perception as someone who works on Mesos, Mesos continues to be popular with large technology companies that don't make their technical investments lightly: Twitter, Apple, Netfli…

Not OP, but I think the perception is that Mesos requires you to roll a lot more of the solution yourself. That's fine if you're a large company who can throw hundreds of developers at your platform, less so if you've got 5 or even 50.
Post reply on HN