Kubermesh: self-hosted/healing/provisioning, partial-mesh network K8s cluster

Post by dominotw » Sun, Aug 20, 2017, 5:45 PM UTC

justicezyx wrote: post #10

dominotw wrote: post #7

kubernetes has so much momentum and developer interest. I rarely see anything about DC/OS which is what they choose at my work last month, apparently its DC/OS is more 'battle tested' in enterprise. I don't think i even want to setup dc/os on my laptop and play with it, seems almost impossible to run mesos on osx . All these articles are making me resent DC/OS :D.

DC/OS was developed with objectives that are not overlapping with K8s'. Namely, they target machine management. While K8s is about container centric job management. Seems subtle, but if you think along these 2 lines, design decisions in 2 systems fall through naturally.

Post by vkat » Sun, Aug 20, 2017, 6:27 PM UTC

dguaraglia wrote: post #5

oarsinsync wrote: post #2

As someone who doesn't know enough about K8s, this looks like an amazingly easy way to get stuck in. Can anyone who has experience with real deployments advise on what pain-points may be encountered by growing something like this further than N nodes (and talk about what N might be?)

I've found a few not-so-obvious pain points on my very limited k8s experience. 1) k8s makes a lot of sense for stateless applications (such as your website) but not so much for stateful applications that require a client to connect to the same container every time (there are ways to do it, but they are a pain in the ass.) 2) Tooling is getting better with time, but it's still pretty green. Packages for your usual orc…

Post by alexnewman » Sun, Aug 20, 2017, 7:28 PM UTC

dguaraglia wrote: post #12

alexnewman wrote: post #8

Earlier quoted context omitted.

1) That's true, but why not support shared state on shared storage. If that's not an option. 2) What exactly do you not like tooling wise. 3) Totally agree. I am actually working on replacing etcd with postgresql though.

Regarding #2, I think it's hard to integrate k8s with the rest of your infrastructure. If you are using, say, a combination of Terraforms + Ansible or Puppet to keep everything in shape, right now you have two options: 1) Write a bunch of bash scripts around kubectl and a bunch of YAML files. While painful, this the way I ended up going for (plus blackbox to GPG encrypt/decrypt secrets on the repo) 2) Try to use your…

Post by eropple » Sun, Aug 20, 2017, 8:18 PM UTC

kuschku wrote: post #16

dguaraglia wrote: post #5

Earlier quoted context omitted.

I've found a few not-so-obvious pain points on my very limited k8s experience. 1) k8s makes a lot of sense for stateless applications (such as your website) but not so much for stateful applications that require a client to connect to the same container every time (there are ways to do it, but they are a pain in the ass.) 2) Tooling is getting better with time, but it's still pretty green. Packages for your usual orc…

> Using their suggested YAML format leads to another problem: there's no easy way to keep secrets outside of the configuration files, unless you build your own process around it Isn’t that what the Secrets resources are for? Just use those, and mount them into your containers in the filesystem or environment variables.

Post by corndoge » Sun, Aug 20, 2017, 9:41 PM UTC

mbryant wrote: post #18

web007 wrote: post #4

This sounds like buzzword bingo. I can't tell without reading the source, is this any different from minikube? I see a mention of NUCs and netboot in the README, so I'm guessing it's slightly more, but not documented.

There's a better explanation of what's going on over here: http://ocadotechnology.com/blog/creating-a-distributed-data-... There's some libvirt stuff, but that's mostly for local testing without having a bunch of physical hardware. The key things I'm trying to do are: - Automatically provision new machines, just by plugging them in. - Don't have a traditional network (i.e. switches)