Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubermesh: self-hosted/healing/provisioning, partial-mesh network K8s cluster

github.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: Kubermesh: self-hosted/healing/provisioning, partial-mesh network K8s cluster

#21

Post by mbryant »

corndoge wrote: post #19
mbryant wrote: post #18

Earlier quoted context omitted.

There's a better explanation of what's going on over here: http://ocadotechnology.com/blog/creating-a-distributed-data-... There's some libvirt stuff, but that's mostly for local testing without having a bunch of physical hardware. The key things I'm trying to do are: - Automatically provision new machines, just by plugging them in. - Don't have a traditional network (i.e. switches)

From the article, you're using Quagga -- are you aware of the active fork FRRouting[0]? It was forked about 8 months ago and is over 3,000 commits ahead since most of the core developers have stopped developing Quagga. [0] https://github.com/FRRouting/frr

Yes, but thanks :)

FRRouting took off after I'd already moved on to other things. I've dropped an issue in the repo so I remember, when I get a chance to work on Kubermesh again

Re: Kubermesh: self-hosted/healing/provisioning, partial-mesh network K8s cluster

#22

Post by SEJeff »

eropple wrote: post #17
kuschku wrote: post #16

Earlier quoted context omitted.

> Using their suggested YAML format leads to another problem: there's no easy way to keep secrets outside of the configuration files, unless you build your own process around it Isn’t that what the Secrets resources are for? Just use those, and mount them into your containers in the filesystem or environment variables.

Kubernetes "secrets" aren't actually...well...secret. They're stored unencrypted in etcd with no ACLs. You must go outside the Kubernetes system to do this. If I control a physical cluster of machines end-to-end, I might consider Kubernetes (because I don't have any better options and doing the work to actually secure it is probably less work than my alternatives), but it's also one of the reasons--though far from th…

Kubernetes 1.7 added the concept of a Provider to encrypt secrets to/from etcd3:

https://docs.google.com/document/d/1lFhPLlvkCo3XFC2xFDPSn0jA...

https://github.com/kubernetes/kubernetes/pull/41939

https://github.com/kubernetes/kubernetes/pull/49350

etc.

There is an implementation to use Hashicorp's Vault to encrypt secrets (using this functionality) which looks quite nice actually:

https://docs.google.com/document/d/15-baW4i7qws1yxxIYjHXqKpk...

Re: Kubermesh: self-hosted/healing/provisioning, partial-mesh network K8s cluster

#23

Post by jzelinskie »

web007 wrote: post #4

This sounds like buzzword bingo. I can't tell without reading the source, is this any different from minikube? I see a mention of NUCs and netboot in the README, so I'm guessing it's slightly more, but not documented.

NUCs are pretty standard for how many people are testing bare metal installations on the cheap. It's what we use in the labs at CoreOS.

This looks like an opinionated installation for an actual bare metal cluster complete with anycast mirrors of popular registries. Minikube is only really good for running in a VM locally.

Re: Kubermesh: self-hosted/healing/provisioning, partial-mesh network K8s cluster

#24

Post by markbnj »

dguaraglia wrote: post #5
oarsinsync wrote: post #2

As someone who doesn't know enough about K8s, this looks like an amazingly easy way to get stuck in. Can anyone who has experience with real deployments advise on what pain-points may be encountered by growing something like this further than N nodes (and talk about what N might be?)

I've found a few not-so-obvious pain points on my very limited k8s experience. 1) k8s makes a lot of sense for stateless applications (such as your website) but not so much for stateful applications that require a client to connect to the same container every time (there are ways to do it, but they are a pain in the ass.) 2) Tooling is getting better with time, but it's still pretty green. Packages for your usual orc…

Regarding CI/CD we use gitlab pipelines to build and push containers and run 'helm upgrade --install' to update charts in the cluster to use the newly pushed images. Works very well and not at all difficult to set up.

Re: Kubermesh: self-hosted/healing/provisioning, partial-mesh network K8s cluster

#25

Post by dghubble »

jzelinskie wrote: post #23
web007 wrote: post #4

This sounds like buzzword bingo. I can't tell without reading the source, is this any different from minikube? I see a mention of NUCs and netboot in the README, so I'm guessing it's slightly more, but not documented.

NUCs are pretty standard for how many people are testing bare metal installations on the cheap. It's what we use in the labs at CoreOS. This looks like an opinionated installation for an actual bare metal cluster complete with anycast mirrors of popular registries. Minikube is only really good for running in a VM locally.

Just to clarify - there are a few NUCs, but its mostly SuperMicros and other vendors where testing and qa are done.
Post reply on HN