Kubernetes 1.6: Multi-user, Multi-workloads at Scale • Hacker News Board

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#21

Post by caio1982 » Tue, Mar 28, 2017, 10:58 PM UTC

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

I don't know if it's the holy grail but it's surely one of the nicest and most impressive piece of tech I've seen since I tried Asterisk 10 years ago and had a long OMG moment myself.

The ecosystem is growing by the day and Kubernetes itself changes a lot very quickly. The pace of its development is sometimes crazy. You might find yourself trying to stay on top of the latest release just to notice there's a new one out of the door and that you will surely want to move to. In this regard I would say to wait a bit, hold your horses and come back to Kubernetes once its release cycle is a bit saner for production environments (my personal estimation is by the end of this year, or earlier than that).

What is missing in my opinion is a better container backend other than Docker (I just find it not enough for more complex scenarios, too bloated, there is the PID-1-hell, etc). With the new container runtime interface I am sure this won't be a problem in the long term.

Also, storage. I mean, the whole user side of setting it up is a breeze. The new release makes it even more simple apparently. However I'm still not very sure how reliable things are here. Maybe it's just me not understanding how storage works in a Kubernetes world... whatever. I know for a fact I am not the only one with this concern though.

What I can tell you is that after seeing a full deployment pipeline in Kubernetes actually working, and seeing all its load balancing and HA and rollout/rollback features, I am not going to stop using it.

My recommendation for starters: the docs are good, read about every Object type and things will eventually "click" in your head. Make sure you understand every bit that is inside a spec file, it will make things easier to understand. Get how ingress/egress work, they can be a pain in Kubernetes sometimes. Try minikube, for Darwin's sake.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#22

Post by caio1982 » Tue, Mar 28, 2017, 11:02 PM UTC

caio1982 wrote: post #21

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

I don't know if it's the holy grail but it's surely one of the nicest and most impressive piece of tech I've seen since I tried Asterisk 10 years ago and had a long OMG moment myself. The ecosystem is growing by the day and Kubernetes itself changes a lot very quickly. The pace of its development is sometimes crazy. You might find yourself trying to stay on top of the latest release just to notice there's a new one o…

Just a clarification: I am not saying Kubernetes will only be good for production envs by the end of the year, what I tried to say is that this year is going to be THE year for Kubernetes and it will see lots of changes that makes it worth to wait a little longer and start with the right release that contains all the nice bits you would expect it to have, say, today.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#23

Post by thesandlord » Tue, Mar 28, 2017, 11:58 PM UTC

tzaman wrote: post #6

One thing that's still missing but would be quite valuable is a good egress approach/configuration. For example, we use an external PostgreSQL provider in which I can configure access IPs. Since K8s nodes (on GKE) get different IPs, I have to be very loose with CIDR selection, which I don't like.

Is IP whitelisting your only option? I really haven't found a good way to pull this off in Kubernetes. You could set up and instance outside the cluster to act as a proxy, but that just feels like a very substandard solution.

On GCP, you can use the SQL Proxy [1] to avoid IP whitelisting or manual SSL setup. Postgres on GCP is still beta, so you probably don't want to run a production DB with it, but hopefully your provider has a similar option.

1: https://cloud.google.com/sql/docs/postgres/sql-proxy

(I work on Google Cloud)

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#24

Post by rodgerd » Wed, Mar 29, 2017, 12:25 AM UTC

newsat13 wrote: post #11

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instru…

> I found kubernetes very hard to learn as well.

This is why we went down the OpenShift root, but there are not shortage of people wrapping Kube in other things, which seems sensible for a lot of use cases.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#25

Post by xur17 » Wed, Mar 29, 2017, 12:55 AM UTC

seeekr wrote: post #20

tzaman wrote: post #6

One thing that's still missing but would be quite valuable is a good egress approach/configuration. For example, we use an external PostgreSQL provider in which I can configure access IPs. Since K8s nodes (on GKE) get different IPs, I have to be very loose with CIDR selection, which I don't like.

Wouldn't The Right Way (TM) be to have a daemon running on the cluster that's watching either your nodes (if you want to allow access from all your nodes) or specific pods and then call your PostgreSQL provider's API to let it know about valid access IPs dynamically?

That's the approach we used when connecting to a legacy Mongo cluster from a GKE cluster. We ran a pod that subscribed to the Kubernetes api, and updated security group rules in AWS as the nodes changed.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#26

Post by marccarre » Wed, Mar 29, 2017, 8:17 AM UTC

caio1982 wrote: post #21

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

I don't know if it's the holy grail but it's surely one of the nicest and most impressive piece of tech I've seen since I tried Asterisk 10 years ago and had a long OMG moment myself. The ecosystem is growing by the day and Kubernetes itself changes a lot very quickly. The pace of its development is sometimes crazy. You might find yourself trying to stay on top of the latest release just to notice there's a new one o…

> What is missing in my opinion is a better container backend other than Docker (I just find it not enough for more complex scenarios, too bloated, there is the PID-1-hell, etc). With the new container runtime interface I am sure this won't be a problem in the long term.

I have only been experimenting a bit with rkt, and I am very new to it, but as far as I understand it should help to address some of the points regarding the container backend, since:

1) it is just a container runtime,

2) doesn't have the client/server architecture that Docker has,

3) has a lightweight systemd starting as PID1 and reaping/adopting children, and

4) works with Kubernetes.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#27

Post by shykes » Wed, Mar 29, 2017, 8:52 AM UTC

caio1982 wrote: post #21

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

I don't know if it's the holy grail but it's surely one of the nicest and most impressive piece of tech I've seen since I tried Asterisk 10 years ago and had a long OMG moment myself. The ecosystem is growing by the day and Kubernetes itself changes a lot very quickly. The pace of its development is sometimes crazy. You might find yourself trying to stay on top of the latest release just to notice there's a new one o…

> What is missing in my opinion is a better container backend other than Docker (I just find it not enough for more complex scenarios, too bloated, there is the PID-1-hell, etc). With the new container runtime interface I am sure this won't be a problem in the long term.

You are in luck. Today Docker donated its low-level runtime containerd to CNCF, and a CRI implementation is on the way with the help of Google.

It fixes all three issues that you listed: it's lightweight, its API gives you complete control of the container primitives, and the PID1 bug is fixed :) It also works well with systend but doesn't require it, which is a nice plus.

https://blog.docker.com/2017/03/containerd-joins-cncf/

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#28

Post by chatmasta » Wed, Mar 29, 2017, 10:04 AM UTC

tzaman wrote: post #6

One thing that's still missing but would be quite valuable is a good egress approach/configuration. For example, we use an external PostgreSQL provider in which I can configure access IPs. Since K8s nodes (on GKE) get different IPs, I have to be very loose with CIDR selection, which I don't like.

This problem could be solved by finer grained control over network routing configuration in general. The problem seems to be that this configuration differs depending on what network virtualization technology/driver you choose, and so implementing it is best left outside the scope of kubernetes.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#29

Post by jondubois » Wed, Mar 29, 2017, 10:54 AM UTC

FooBarWidget wrote: post #8

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big inform…

I think it's the holy grail for stateless services. For stateful systems though, (that need to persist data) things can be tricky. Maybe the improved node affinity features in this release will improve things but I still feel limited because I can't seem to find a way to permanently bind a Pod to a specific host's file system. They keep pushing for NFS or other complex network file systems that add huge network I/O overheads instead of letting you deal with distributed storage across multiple hosts yourself (which would work better with most new database engines that have built-in cluster support). Last time I checked, you COULD bind a Pod to the underlying host's file system but the documentation says that this is only for debugging purposes and not suitable for production.

Re: Kubernetes 1.6: Multi-user, Multi-workloads at Scale

#30

Post by stephane-klein » Wed, Mar 29, 2017, 11:28 AM UTC

kt9 wrote: post #16

newsat13 wrote: post #11

Earlier quoted context omitted.

Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instru…

Checkout Distelli ( https://www.distelli.com ). UI and pipelines that make using Kubernetes easier. Tons of features coming soon. I can also help you with Docker / Kubernetes even if you decide not to buy anything from us - happy to help. Email is in my profile. Disclaimer: I'm the founder at Distelli.

I think that Distelli is like Build in OpenShift (https://docs.openshift.org/latest/architecture/core_concepts...), isn't it ?