Kubernetes 1.6: Multi-user, Multi-workloads at Scale • Hacker News Board

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#11

Post by newsat13 » Tue, Mar 28, 2017, 9:38 PM UTC

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instructions to follow on a wiki page.

I have also found kubernetes labor hard to find. The ones I found were way too expensive for me (upwards of 150k USD).

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#12

Post by dluc » Tue, Mar 28, 2017, 9:39 PM UTC

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Having worked mostly with mesos and k8s, I found k8s configuration superior, e.g. how one can import secrets, config files, and more importantly set up the network without address translation or port forwarding. Tooling seems OK with both, and I agree one needs to spend some time to get familiar with the CLI and nomenclature, IMHO because both are quite flexible and powerful.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#13

Post by Kubuxu » Tue, Mar 28, 2017, 9:40 PM UTC

newsat13 wrote: post #11

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instru…

Check out Nomad, I have no personal experience with it yet but it seems the best candidate so far for what we are planning in terms of docker deployment and so on.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#14

Post by FooBarWidget » Tue, Mar 28, 2017, 9:42 PM UTC

newsat13 wrote: post #11

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instru…

Have you tried kontena? They aim to be a container orchestrator designed for developer happiness and ux. Certainly looks a lot simpler.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#15

Post by Agrosis » Tue, Mar 28, 2017, 9:45 PM UTC

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

There are definitely a lot of moving parts in Kubernetes just to get it running and it was a big hurdle to get everything set up (I'm running a vanilla cluster in Vagrant).

The documentation definitely covers a lot, but I found myself hunting issues down for trying to get something to work like getting k8s to schedule pods across a flannel overlay network and making sure traffic is being routed properly. Of course, you don't have to worry about any of this if you use GKE or another cloud provider.

Once I got that set up, everything worked beautifully. I can just write a config and apply it to the cluster and services are deployed and scaled and whatnot. Treating your cluster as a pool of resources is pretty powerful. There are a lot of Kubernetes API objects (Deployments, Stateful Sets, Daemon Sets, ConfigMaps, Secrets), but everything has a purpose and Kubernetes makes everything come together nicely.

The Kubernetes Slack channel was really helpful for me as well, there's a lot of nice and friendly people there willing to answer your questions and help you debug your setup.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#16

Post by kt9 » Tue, Mar 28, 2017, 9:46 PM UTC

newsat13 wrote: post #11

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instru…

Checkout Distelli (https://www.distelli.com). UI and pipelines that make using Kubernetes easier. Tons of features coming soon.

I can also help you with Docker / Kubernetes even if you decide not to buy anything from us - happy to help. Email is in my profile.

Disclaimer: I'm the founder at Distelli.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#17

Post by roque » Tue, Mar 28, 2017, 10:11 PM UTC

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Yes, it is the holy grail. Yes, the documentation is a bit spotty in some cases. We've been using kubernetes in GCE for about 1 year. We run both interactive services and batch jobs in 2 clusters: dev and production. The production cluster runs fully with declarative configs (we have a jenkins instance that performs the deployments of committed k8s configs). There are a couple of features that we are missing: (pod anti-affinity now in 1.6) and better support for storage. But in general kubernetes has the right concepts for running services, imho. We have 0 ops people. Our clusters run on large VMs with 10s of PODs in the same VM. We can get high cpu/mem utilization; resilience to failures; logging; monitoring (via prometheus - GCE /stackdriver is awful). I don't think there is anything out there that comes close in terms of supporting the right abstractions and being ready to use in production.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#18

Post by jdc0589 » Tue, Mar 28, 2017, 10:15 PM UTC

newsat13 wrote: post #11

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instru…

there's definitely a lot to it...I'm still in the phase where I'm a bit overwhelmed with trying to learn everything. As far as setup/ops goes, definitely check out https://github.com/kubernetes/kops if you are on AWS; it's not perfect, but it made some stuff way easier in the early stages for me.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#19

Post by jdc0589 » Tue, Mar 28, 2017, 10:21 PM UTC

roque wrote: post #17

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Yes, it is the holy grail. Yes, the documentation is a bit spotty in some cases. We've been using kubernetes in GCE for about 1 year. We run both interactive services and batch jobs in 2 clusters: dev and production. The production cluster runs fully with declarative configs (we have a jenkins instance that performs the deployments of committed k8s configs). There are a couple of features that we are missing: (pod an…

we aren't anywhere close to using it in production yet, but my impression so far is to agree with you. Every single use case or weird requirement I've made up so far has had an answer, or had a solution in at least the proposal phase.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#20

Post by seeekr » Tue, Mar 28, 2017, 10:53 PM UTC

tzaman wrote: post #6

One thing that's still missing but would be quite valuable is a good egress approach/configuration. For example, we use an external PostgreSQL provider in which I can configure access IPs. Since K8s nodes (on GKE) get different IPs, I have to be very loose with CIDR selection, which I don't like.

Wouldn't The Right Way (TM) be to have a daemon running on the cluster that's watching either your nodes (if you want to allow access from all your nodes) or specific pods and then call your PostgreSQL provider's API to let it know about valid access IPs dynamically?