Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes at GitHub

githubengineering.com

Post reply on HN
Print view View original

21–30 of 142 posts

Re: Kubernetes at GitHub

#21

Post by awill »

deadmik3 wrote: post #7
DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

I'd recommend looking into openshift. it's basically kubectl + cool deployment features. there's also free, paid, and dedicated online hosted options. disclaimer: I work on openshift

You had a big announcement about openshift.io. Everyone on HN signed up, but it's been months, but I'm still 'awaiting approval'

Have you let anyone in? What's the value in all the marketing hype if you then don't let people in.

Re: Kubernetes at GitHub

#22

Post by sytse »

lobster_johnson wrote: post #18

I'd be interested in hearing what kind of autoscaling system they use for their Ruby pods. We're running a few (legacy — we're moving to Go) Ruby apps in production on Kubernetes. We're using Puma, which is very similar to Unicorn, and it's unclear what the optimal strategy here is. I've not benchmarked this in any systematic way. For example, in theory you could make a single deployment run a single Unicorn worker,…

At GitLab we recommend to use CPU cores + 1 as the number of unicorn workers https://docs.gitlab.com/ce/install/requirements.html#unicorn...

Re: Kubernetes at GitHub

#23

Post by sytse »

DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

If you're using our GitLab consider using Auto Deploy. Our CTO recently made a quick start guide for it https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.h...

Re: Kubernetes at GitHub

#24

Post by AaronBBrown »

cyphar wrote: post #17
shock wrote: post #8

> During this migration, we encountered an issue that persists to this day: during times of high load and/or high rates of container churn, some of our Kubernetes nodes will kernel panic and reboot. Considering that Kubernetes doesn't modify the kernel, this issue sounds like is present in mainline and kernel devs should be involved.

I would be interested to know what storage driver they're using for their nodes. High container churn puts a lot of stress on the VFS subsystem of Linux, and we've seen cases where customers have trigger lots of mount/umounts which results in filesystems causing panics. At SUSE, we do have some kernel devs debugging the issues, but the workaround is almost always "rate limit all the things". There are a few other ker…

Hey, this is Aaron from GitHub. We're using devicemapper w/ LVM backed pools. Would love to hear about your experience there. We definitely see this problem during periods of high container churn.

Re: Kubernetes at GitHub

#25

Post by lobster_johnson »

sytse wrote: post #22
lobster_johnson wrote: post #18

I'd be interested in hearing what kind of autoscaling system they use for their Ruby pods. We're running a few (legacy — we're moving to Go) Ruby apps in production on Kubernetes. We're using Puma, which is very similar to Unicorn, and it's unclear what the optimal strategy here is. I've not benchmarked this in any systematic way. For example, in theory you could make a single deployment run a single Unicorn worker,…

At GitLab we recommend to use CPU cores + 1 as the number of unicorn workers https://docs.gitlab.com/ce/install/requirements.html#unicorn...

How do you configure that? A pod doesn't know what machine it's running on ahead of time. You can create nodepools and use node selectors to pin the pod to that nodepool, but I'm not sure I love the idea.

Re: Kubernetes at GitHub

#26

Post by majewsky »

DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

At SAP, we're using Helm [1] to deploy OpenStack (plus a bunch of extra services like a custom Rails-based dashboard [2]) on baremetal Kubernetes. For image building, testing and deployment, we use Concourse CI [3], and OpenStack assets (like service users, service projects and roles) are deployed with a custom Kubernetes operator [4].

Our charts are at [5] if you want to take a look.

[1] https://github.com/kubernetes/helm

[2] https://github.com/sapcc/elektra

[3] https://concourse.ci

[4] https://github.com/sapcc/kubernetes-operators in the "openstack-operator" directory

[5] https://github.com/sapcc/openstack-helm and https://github.com/sapcc/helm-charts (two different repos since we're in the middle of modularizing the original monolithic OpenStack chart in the first repo into per-service charts in the second one)

Re: Kubernetes at GitHub

#27

Post by majewsky »

ben_jones wrote: post #13
Yeroc wrote: post #9

One thing I would have liked to have seen addressed in the article is whether the new architecture requires additional hardware (presumably) to operate and if so how much more.

I've only dabbled in K8s and it strikes me that using it in production is a long term investment and, as it stands currently, a long term project to implement properly. You'll want to do exactly what Github did: setup a "review lab" or similarly comprehensive dev and test environment until you are absolutely comfortable with it in production. This will lead to the provisioning (and cost) of quite a bit of hardware -…

I agree very much, and I'd like to add one point: When you build a lab environment for testing Kubernetes deployments (and verifying Kubernetes upgrades), make sure it's on the same hardware as your production environment.

When my team did the first Kubernetes deployment, we made the mistake of building a lab environment that did not match the anticipated production environment. (Two reasons: The BOM for the production environment was not yet decided upon at that time, and the lab was frankensteined together by taking hardware out of existing labs.) We learned the hard way that, just because the Kubernetes upgrade worked in the lab, it need not work on the production hardware.

Right now, we're stuck on last year's (i.e., ancient) Kubernetes 1.4 release because no one dares to upgrade production. (There's light at the end of the tunnel, though. A new lab is being built up in the datacenter around now.)

Re: Kubernetes at GitHub

#28

Post by Snappy »

sytse wrote: post #23
DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

If you're using our GitLab consider using Auto Deploy. Our CTO recently made a quick start guide for it https://docs.gitlab.com/ee/ci/autodeploy/quick_start_guide.h...

FWIW, the next version of GitLab's Auto Deploy will use Helm under the hood (and let you bring-your-own-chart).

Re: Kubernetes at GitHub

#29

Post by cyphar »

AaronBBrown wrote: post #24
cyphar wrote: post #17

Earlier quoted context omitted.

I would be interested to know what storage driver they're using for their nodes. High container churn puts a lot of stress on the VFS subsystem of Linux, and we've seen cases where customers have trigger lots of mount/umounts which results in filesystems causing panics. At SUSE, we do have some kernel devs debugging the issues, but the workaround is almost always "rate limit all the things". There are a few other ker…

Hey, this is Aaron from GitHub. We're using devicemapper w/ LVM backed pools. Would love to hear about your experience there. We definitely see this problem during periods of high container churn.

That's funny, we have an internal bug open right now about a kernel panics that happen with devicemapper (with XFS as the base filesystem). We found that the issue was exacerbated if you used loopback devices, but on paper it should still happen in non-loopback mode (the current theory is that it's a bug in XFS). Our kernel team is still investigating the issue, but they cannot seem to reproduce the issue with direct-lvm (and loop-lvm is inconsistent in reproducing the issue).

If you can consistently reproduce the issue, would you mind providing the backtrace and/or coredump? Is it possible for you to reproduce the issue on a machine without needing to be hit by GitHub-levels of traffic, and if so can you provide said reproducer?

For reference, our backtraces show that the kernel dies at Xfs_vm_writepage. Though of course different kernel versions may have varying backtraces.

You can reach me on the email in my profile, or asarai(at)suse.com.

Re: Kubernetes at GitHub

#30

Post by Snappy »

lobster_johnson wrote: post #19
DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

We wrote an internal tool that wraps Helm and GPG. But we're really using Helm as a glorified templating system; since we deploy from git, Helm's release management is useless to us, and is even somewhat in the way. We might decide to drop Helm at some point, I think.

That's a good point. We (GitLab) use it mostly as a templating system as well. It's a step up from piping `sed` output to `kubectl`. But we have our own tools for managing redeploys and rollbacks.
Post reply on HN