Kubernetes 1.3 released

so true, the self service aspect is indeed amazing. once developers or qa people grok the concepts and api it can do wonders to your productivity.

also, working with k8s will probably spoil you, it's pretty annoying to "go back" to other environments, where you're confronted with problems which would be effortlessly solvable in kubernetes.

I'm still sitting on the sidelines waiting for the easy to install, better documented for AWS version. It's also a bit unclear as to why we are talking federation and master/slave in 2016; other systems are using raft and gossip protocols to build masterless management clusters..

Watching issues like https://github.com/kubernetes/kubernetes/issues/23478 , and https://github.com/kubernetes/kubernetes/issues/23174 .. I'm not super interested in "kicking the tires"; I'm evaluating replacing all our environment automation with a version built around Kubernetes. Easy-up scripts that hide a ton of nasty complexity won't do the trick.

Following the issues I'm getting the impression that too much effort is being put into CM style tools vs making the underlying components more friendly to setup and manage. Did anyone see how easy it is to get the new Docker orchestration running?

Then there is the AWS integration documentation.. I'm following the hidden aws_under_the_hood.md updates, but I'm still left with loads of questions; like how do I control the created ELB's configuration(cross zone load balancing, draining, timeouts,etc)?

I re-evaluate after every update and there are some really nice features being added in, but at the end of the day ECS is looking more and more the direction to go for us. Sure, it's lacking a ton of features compared to Kubernetes and it's nigh but impossible to get any sort of information about roadmaps out of Amazon... But it's very clear how it integrates with ELB and how to manage the configuration of every underlying service. It also doesn't require extra resources(service or human) to setup and manage the scheduler.

It's funny how words can be played. The Kubernetes "master" is a set of 1 or more machines that run the API server and associated control logic. This is exactly what systems like Docker swarm do, but they wrap it in terms like RAFT and gossip that make people weak in the knees. Kubernetes has RAFT in the form of the storage API (etcd). This is a model that has PROVEN to work well, and to scale well beyond what almost anyone will need.

"Federation" in this context is across clusters, which is not something other systems really do much of, yet. You certainly don't want to gossip on this layer.

"evaluating replacing" really does imply "kicking the tires". Put another way - how much energy are you willing to invest in the early stages of your evaluation? If a "real" cluster took 3 person-days to set up, but a "quick" cluster took 10 person-minutes, would you use the quick one for the initial eval? Feedback we have gotten repeatedly was "it is too hard to set up a real cluster when I don't even know if I want it".

There are a bunch of facets of streamlining that we're working on right now, but they are all serving the purposes of reducing initial investment and increasing transparency.

> how easy it is to get the new Docker orchestration running

This is exactly my point above. You don't think that their demos give you a fully operational, secured, optimized cluster with best-perf networking, storage, load-balancing etc, do you? Of course not. It sets up the "kick the tires" cluster.

As for AWS - it is something we will keep working on. We know our docs here are not great. We sure could use help tidying them up and making them better. We just BURIED is things to do.

Thanks for the feedback, truly.

One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work.

All you need to do, in broad strokes, is:

* Set up a VPC. Defaults work.

* Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs.

* Install Kubernetes from binary packages. I've been using Kismatic's Debian/Ubuntu packages [2], which are nice.

* Install Docker >= 1.9 * Install etcd.

* Make sure your AWS instance has a sane MTU ("sudo ifconfig eth0 mtu 1500"). AWS uses jumbo frames by default [3], which I found does not work with Docker Hub (even though it's also on AWS).

* Edit /etc/default/docker to disable its iptables magic and use the Kubernetes bridge, which Kubelet will eventually create for you on startup:

   DOCKER_OPTS="--iptables=false --ip-masq=false --bridge=cbr0"

* Decide which CIDR ranges to use for pods and services. You can carve a /24 from your VPC subnet for each. They have to be non-overlapping ranges.

* Edit the /etc/default/kube* configs to set DAEMON_ARGS in each. Read the help page for each daemon to see what flags they take. Most have sane defaults or are ignorable, but you'll need some specific ones [4].

* Start etcd, Docker and all the Kubernetes daemons.

* Verify it's working with something like: kubectl run test --image=dockercloud/hello-world

Unless I'm forgetting something, that's basically it for one master node. For multiple nodes, you'll have to run Kubelet on each. You can run as many masters (kube-apiserver) as you want, and they'll use etcd leases to ensure that only one is active.

[1] https://gist.github.com/atombender/3f9ba857590ea98d18163e983...

[2] http://repos.kismatic.com/debian/

[3] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_m...

[4] https://gist.github.com/atombender/e72c2acc2d30b0965543273a2...

I would consider "kicking the tires" actually running up a cluster and playing with it. One can also evaluate by reading documentation and others reports of issues to look for show-stopping problems. For instance, a couple releases ago there was not multi-AZ support. The word on the street at that time was to create multiple clusters and do higher level orchestration across them.. That was a no-go for us; no need to "kick the tires".

Whatever you may think of my level of knowledge or weak knees for consensus and gossip protocols, these problems(perceived or otherwise) with setup, documentation, and management seem pretty widely reported.

EDIT: I hope this doesn't sound too negative. Kubernetes IS getting better all the time. I only write this to give a perspective from somebody who would like use Kubernetes but has reason for pause. Our requirements are likely not standard; our internal bar for automation and ease of use is quite high. We essentially have an internal, hand-rolled, docker-based PaaS with support for ad-hoc environment creation(not just staging/prod). We would like to move away from holding the bag on our hand-rolled stuff and adopt a scheduler :) Deciding to pull the trigger on any scheduler though would be committing us to a rather large amount of integration effort to reach a parity that doesn't seem riddled with regressions over the current solution.