Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes at GitHub

githubengineering.com

Post reply on HN
Print view View original

61–70 of 142 posts

Re: Kubernetes at GitHub

#61

Post by brown9-2 »

spmurrayzzz wrote: post #57
Alan01252 wrote: post #44

I'm curious to what this means for the existing puppet code base, is it now irrelevant, or are there still usages for it in the k8s world?

They could easily still use standalone puppet to handle the config management for individual container images. I currently do this with salt-minion. It reduces the burden on the Dockerfile itself, and lets you embrace a declarative configuration state at build time.

This introduces a dependency on which pod runs on which host, unless you have puppet write the config for every service to every host.

People tend to use Puppet less to configure their applications as they move into containers, as a configuration change can just be made by rolling out a new image.

Re: Kubernetes at GitHub

#62

Post by ninkendo »

AaronBBrown wrote: post #24
cyphar wrote: post #17

Earlier quoted context omitted.

I would be interested to know what storage driver they're using for their nodes. High container churn puts a lot of stress on the VFS subsystem of Linux, and we've seen cases where customers have trigger lots of mount/umounts which results in filesystems causing panics. At SUSE, we do have some kernel devs debugging the issues, but the workaround is almost always "rate limit all the things". There are a few other ker…

Hey, this is Aaron from GitHub. We're using devicemapper w/ LVM backed pools. Would love to hear about your experience there. We definitely see this problem during periods of high container churn.

Any particular reason you didn't choose something like overlay2?

Re: Kubernetes at GitHub

#63

Post by invisible »

DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

We are using ecfg (from Shopify) and Jenkins+kubectl. We use ansible for a couple of things but it's largely only because of some parameters of our architecture for a legacy monolith.

Re: Kubernetes at GitHub

#64

Post by ninkendo »

spmurrayzzz wrote: post #57
Alan01252 wrote: post #44

I'm curious to what this means for the existing puppet code base, is it now irrelevant, or are there still usages for it in the k8s world?

They could easily still use standalone puppet to handle the config management for individual container images. I currently do this with salt-minion. It reduces the burden on the Dockerfile itself, and lets you embrace a declarative configuration state at build time.

It definitely seems like the wrong approach to me to have puppet manage your base images. They're not VM's, they shouldn't have multiple services, they shouldn't require any complex configuration management, they should just be the minimum requirements to support your application's local runtime dependencies, and that's it.

From previous experience migrating from a puppet setup to one that used containers, puppet's vestigial use case ends up being to get the orchestration control plane itself setup (ie. kubernetes, networking configs, etc) and that's about it.

Re: Kubernetes at GitHub

#65

Post by bpineau »

AaronBBrown wrote: post #24
cyphar wrote: post #17

Earlier quoted context omitted.

I would be interested to know what storage driver they're using for their nodes. High container churn puts a lot of stress on the VFS subsystem of Linux, and we've seen cases where customers have trigger lots of mount/umounts which results in filesystems causing panics. At SUSE, we do have some kernel devs debugging the issues, but the workaround is almost always "rate limit all the things". There are a few other ker…

Hey, this is Aaron from GitHub. We're using devicemapper w/ LVM backed pools. Would love to hear about your experience there. We definitely see this problem during periods of high container churn.

A known (and now fixed) kernel issue affects the scheduler and cgroups subsystem, triggering crashs under kubernetes load (fixed by 754bd598be9bbc9 and 094f469172e00d). The fix was merged in Linux 4.7 (and backported to -stable, in 4.4.70). So if you run an older kernel, maybe you are hit by this?

Re: Kubernetes at GitHub

#68

Post by Omnipresent »

dookahku wrote: post #66

Any favorted training for learning Kubernetes? I found this one so far: https://classroom.udacity.com/courses/ud615 But any extra courses/trainings is always appreciated

I'm interested in this as well. I found Hepito https://www.heptio.com/support-services-and-training

Re: Kubernetes at GitHub

#69

Post by drdaeman »

How hard (and how realistic) it is to actually get a reasonable understanding (and then stay up-to-date) with Kubernetes internals? Is there any go-to reading material?

We had ran another large-footprint container management system (not K8s, but also popular), and when its DNS component started to eat all the CPU on all nodes, best I was able to do fast,was just scrapping the whole thing and quickly replacing it with some quick-and-dirty Compose files and manual networking. At least, we were back to normal in an hour or so. Obvious steps (recreating nodes) failed, logs looked perfectly normal, quick strace/ltrace gave no insights, and trying to debug the problem in detail would've taken more time.

But that was only possible because all we ran was small 2.5-node system, not even a proper full HA or anything. And it had resembled Compose close enough.

Since then I'm really wary about using larger black boxes for critical parts. Just Linux kernel and Docker can bring enough headache, and K8s on top of this looks terrifying. Simplicity has value. GitHub can afford to deal with a lot of complexity, but a tiny startup probably can't.

Or am I just unnecessarily scaring myself?

Re: Kubernetes at GitHub

#70

Post by joombaga »

erulabs wrote: post #67

If you're running Kube on AWS, make sure you install the proper drivers! For Ubuntu, that's the `linux-aws` apt package. https://github.com/kubernetes/kops/issues/1558 Missing ENA and ixgbevf can be a real performance killer!

Is this used by vanilla docker / ECS, or just k8s?
Post reply on HN