Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes: The Future of Deployment

bashton.com

Post reply on HN
Print view View original

41–50 of 80 posts

Re: Kubernetes: The Future of Deployment

#41

Post by Hortinstein »

iyn wrote: post #9
SEJeff wrote: post #5

Earlier quoted context omitted.

Kubernetes (k8s) makes for an amazing developer story. Mesos is much more bare metal, but the scheduler scales a loooot better than the still relatively immature k8s scheduling component. One of the original authors of mesos wrote a paper on scheduling: https://www.cs.berkeley.edu/~alig/papers/drf.pdf . Mesos is one of the first "two level" schedulers. I very highly recommend that you also read this article for an id…

Can you suggest best resources (text/video) for learning about Kubernetes & Mesos? I use Docker & CoreOS all the time (love it) and I'm always trying to improve/learn something new. Can you tell how do you use Apache Aurora? What other interesting projects are worth learning about?

how do you currently do service discovery w/ Docker and CoreOS without MESOS or Kubernetes?

Re: Kubernetes: The Future of Deployment

#42

Post by SEJeff »

lobster_johnson wrote: post #13
SEJeff wrote: post #5

Earlier quoted context omitted.

Kubernetes (k8s) makes for an amazing developer story. Mesos is much more bare metal, but the scheduler scales a loooot better than the still relatively immature k8s scheduling component. One of the original authors of mesos wrote a paper on scheduling: https://www.cs.berkeley.edu/~alig/papers/drf.pdf . Mesos is one of the first "two level" schedulers. I very highly recommend that you also read this article for an id…

Thanks for the explanation. Sounds like Kubernetes should work just fine for small ( I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexi…

Zookeeper is a hog, no one will disagree. But etcd is a very awesome yet very new technology. Even the protocol it implements (which is awesome), raft, is very new as a distributed consensus protocol. I'm not in any remote means throwing cold water on k8s, it is fantastic stuff. I only know that there are very very large production mesos clusters today, and the same can not be said (yet) for k8s. Read those two links I posted in the parent though if you have the time. It will make a ton more sense.

That being said, you k8s is sexy stuff, it just ties you to docker, and I believe soon to be rocket. When I first started evaluating both (around docker 1.2.x), docker was not super viable and was pretty buggy. With 1.6.x and newer, most of my original concerns cease to matter. They are both excellent technologies, use whatever works for your environment.

Re: Kubernetes: The Future of Deployment

#43

Post by cthalupa »

josephjacks wrote: post #24
jaytaylor wrote: post #8

Earlier quoted context omitted.

Thanks for sharing all that terrific information! The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max. Source: One of the core k8s developers I met at a CoreOS meetup in SF earlier this year. They said if I needed to go beyond 500 nodes that I should probably look at something else.

This is categorically incorrect. K8S will ultimately scale to N number of nodes. Within 2015, it will scale to 1K+ nodes, as per the roadmap. Being modeled after Google's Borg system, I encourage curious/interested folks to read at the recent Borg paper [0] which also outlines lessons learned in running Borg at Google for nearly 15 years and managing many millions of machines. [0] http://research.google.com/pubs/pub4…

Calling that categorically incorrect is pretty disingenuous when we have Google engineers who are working on the project saying that it currently supports 100 nodes with ease, and that they /expect/ it to handle more in the (very near) future.

It might not be correct for much longer, but if it is the case now, how can you say it's categorically incorrect?

Re: Kubernetes: The Future of Deployment

#44

Post by josephjacks »

cthalupa wrote: post #43
josephjacks wrote: post #24

Earlier quoted context omitted.

This is categorically incorrect. K8S will ultimately scale to N number of nodes. Within 2015, it will scale to 1K+ nodes, as per the roadmap. Being modeled after Google's Borg system, I encourage curious/interested folks to read at the recent Borg paper [0] which also outlines lessons learned in running Borg at Google for nearly 15 years and managing many millions of machines. [0] http://research.google.com/pubs/pub4…

Calling that categorically incorrect is pretty disingenuous when we have Google engineers who are working on the project saying that it currently supports 100 nodes with ease, and that they /expect/ it to handle more in the (very near) future. It might not be correct for much longer, but if it is the case now, how can you say it's categorically incorrect?

To clarify, this entire statement is categorically incorrect: "The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max."

Re: Kubernetes: The Future of Deployment

#45

Post by jganetsk »

brendandburns wrote: post #34
xorcist wrote: post #30

Earlier quoted context omitted.

Google runs docker in a VM? Why would you do that? I understand you might want to play with it in a lab environment, but in production at scale (especially Google-scale) sounds very strange.

Users of the Google Cloud run Docker in VMs, since VMs are what the Google Cloud Platform sells. (as does every public cloud provider [e.g. AWS]) For now, VMs are required to ensure a security barrier between different user's containers on the same physical machine. See some of Dan Walsh's posts on the subject (e.g. https://opensource.com/business/14/9/security-for-docker ) for more context.

Google Container Engine runs containers that are in Docker format. The user does not have to deal with Docker or a VM.

https://cloud.google.com/container-engine/

There's also Amazon EC2 Container Service

http://aws.amazon.com/ecs/details/

So Google and Amazon don't just sell VMs. They sell "CMs" as well (Container Machines).

Re: Kubernetes: The Future of Deployment

#46

Post by iyn »

Hortinstein wrote: post #41
iyn wrote: post #9

Earlier quoted context omitted.

Can you suggest best resources (text/video) for learning about Kubernetes & Mesos? I use Docker & CoreOS all the time (love it) and I'm always trying to improve/learn something new. Can you tell how do you use Apache Aurora? What other interesting projects are worth learning about?

how do you currently do service discovery w/ Docker and CoreOS without MESOS or Kubernetes?

Honest answer: I don't. In short: most of the things have fixed config that is loaded into etcd cluster and different services in Docker containers use it to communicate with other containers/services (something like {rabbitmq_host: "host address"}. In the project I'm working on right now I have just 10 boxes which will probably grow to 20-30 in the coming months. It's nothing, I know, and as you can tell from the hacky nature of my setup I'm learning as I go about this, but I'm trying to incrementally improve different parts. Something like Kubernetes/Mesos seems like a next step.

Re: Kubernetes: The Future of Deployment

#47

Post by gobengo »

jganetsk wrote: post #45
brendandburns wrote: post #34

Earlier quoted context omitted.

Users of the Google Cloud run Docker in VMs, since VMs are what the Google Cloud Platform sells. (as does every public cloud provider [e.g. AWS]) For now, VMs are required to ensure a security barrier between different user's containers on the same physical machine. See some of Dan Walsh's posts on the subject (e.g. https://opensource.com/business/14/9/security-for-docker ) for more context.

Google Container Engine runs containers that are in Docker format. The user does not have to deal with Docker or a VM. https://cloud.google.com/container-engine/ There's also Amazon EC2 Container Service http://aws.amazon.com/ecs/details/ So Google and Amazon don't just sell VMs. They sell "CMs" as well (Container Machines).

It's most likely that even the "CM"s from both providers are actually Virtual Machines running on a hypervisor running on bare metal. You just can't tell and don't need to care (for most workloads).

Re: Kubernetes: The Future of Deployment

#48

Post by shykes »

iyn wrote: post #46
Hortinstein wrote: post #41

Earlier quoted context omitted.

how do you currently do service discovery w/ Docker and CoreOS without MESOS or Kubernetes?

Honest answer: I don't. In short: most of the things have fixed config that is loaded into etcd cluster and different services in Docker containers use it to communicate with other containers/services (something like {rabbitmq_host: "host address"}. In the project I'm working on right now I have just 10 boxes which will probably grow to 20-30 in the coming months. It's nothing, I know, and as you can tell from the ha…

In case you're interested, the next version of Docker (1.7) supports multi-host networking and dynamic service discovery out of the box.

The whole thing is pluggable and can use various distributed state backends (etcd, zookeeper etc) or IP connectivity backends (veth, macvlan, vxlan, openvpn etc) without changing your application. Service discovery uses DNS so you don't need to modify your application to take advantage of it. It's probably the most significant change to Docker in the last year.

This will make the integration with Kubernetes smoother. Currently Google is forced to rip out Docker's native networking stack because it is not flexible enough for their opinionated networking model. This causes many Docker applications to break in Kubernetes today. That problem should go away with Docker 1.7+ because Google-style networking can be expressed as a Docker plugin, which Kubernetes can load programmatically as part of its orchestration. An added benefit is that you can augment Docker with Google-style networking even if you use Kubernetes competitors like Mesos, Swarm, Cloudfoundry etc.

(EDIT added details more relevant to Kubernetes)

Re: Kubernetes: The Future of Deployment

#49

Post by falcolas »

gobengo wrote: post #47
jganetsk wrote: post #45

Earlier quoted context omitted.

Google Container Engine runs containers that are in Docker format. The user does not have to deal with Docker or a VM. https://cloud.google.com/container-engine/ There's also Amazon EC2 Container Service http://aws.amazon.com/ecs/details/ So Google and Amazon don't just sell VMs. They sell "CMs" as well (Container Machines).

It's most likely that even the "CM"s from both providers are actually Virtual Machines running on a hypervisor running on bare metal. You just can't tell and don't need to care (for most workloads).

Yup, you can even SSH to them and poke around yourself.

Re: Kubernetes: The Future of Deployment

#50

Post by roasbeef »

falcolas wrote: post #23

Worth remembering, Kubernetes was built to Google's needs, and Google runs with a shared network space on any given VM and assigns an entire /24 to the VM running docker. Each container gets one of those addresses. [1] This probably won't work for everyone - be sure to read into the fine grained details before drinking the koolade. They're also at least two build versions behind Docker.[2] [1] https://github.com/Goog…

Indeed the networking requirements of Kubernetes are very different from that of Docker.

However, it isn't too difficult to setup a Layer 2 unification overlay network. I've had success with flannel[1], it's pretty easy to set up and supports VxLAN.

[1] https://github.com/coreos/flannel

Post reply on HN