Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes at GitHub

githubengineering.com

Post reply on HN
Print view View original

31–40 of 142 posts

Re: Kubernetes at GitHub

#31

Post by nikon »

foxylion wrote: post #6
DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

We also did some evaluation and then decided to stick to KISS an chose kubectl commands combined with cat and kexpand. Really simple approach to allow dynamic kubernetes deployments. Example command can be cat service.yml | kexpand expand -v image-tag=git-135afed4 | kubectl apply -f - The service.yml contains the full deployment configuration, service definition and ingress rules. So this works without preconfiguring…

I do something similar, but envsubst does the job.

Re: Kubernetes at GitHub

#32

Post by yebyen »

gtirloni wrote: post #16
yebyen wrote: post #14

Earlier quoted context omitted.

You know it's open source, right? I have zero guarantees that any of my open source projects that I use for business critical infrastructure aren't going to pack up shop and quit maintaining their stuff tomorrow. You should know how your infrastructure works well enough to maintain it for yourself. I (personally) will be maintaining this one in the future, if necessary! We're working it out now. What do you mean by "…

Unfortunately, neither it being open source nor its technical prowess is reason sufficient for some people. That's a simplistic analysis. Most people using Linux don't know it the kernel code "well enough" to maintain it in face of hardware changes and other external requirements. I don't mean anything by "strong suspicions", you do: "I have a strong suspicion that Deis will live on after March under stewardship of n…

OK, fair. It's still worth a look! That was my point. If you compare it to other alternatives and find it to be the best thing, it would be a shame to put it on the shelf instead when there are no alternatives that are as technically strong.

If you don't look at it, you have no basis to compare it to the alternatives that you're evaluating. I would rather pick a really good solution than a supported one. If I have to pick a supported solution, then I would rather at minimum compare it to a good one, so I can be honest about what shortcomings it has. It is telling that most of the recommendations in this thread are Helm.

Helm is great, but it ain't no Deis.

Deis Workflow is rock-solid and worth evaluating, even if support is a requirement for you. Other competing solutions are not as good. I don't even know any other that are really comparable. Maybe OpenShift, but it is not "really K8S"

Re: Kubernetes at GitHub

#33

Post by sandGorgon »

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services.

Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system .

This is the single biggest reason why using Docker Swarm is so pleasant.

Re: Kubernetes at GitHub

#34

Post by joshuatalb »

ben_jones wrote: post #13
Yeroc wrote: post #9

One thing I would have liked to have seen addressed in the article is whether the new architecture requires additional hardware (presumably) to operate and if so how much more.

I've only dabbled in K8s and it strikes me that using it in production is a long term investment and, as it stands currently, a long term project to implement properly. You'll want to do exactly what Github did: setup a "review lab" or similarly comprehensive dev and test environment until you are absolutely comfortable with it in production. This will lead to the provisioning (and cost) of quite a bit of hardware -…

If you're on AWS you can use kops (https://github.com/kubernetes/kops) to significantly reduce the amount of time to get a cluster up. It took me about 1hr to get a basic k8s cluster up and running with it.

Re: Kubernetes at GitHub

#35

Post by nikon »

bryanlarsen wrote: post #10
DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

We're using helm, but that was chosen mostly based on gut feel. It's an official project, and has momentum. We didn't want to spend too much time choosing a tool until we knew our requirements, and we don't really have a firm grasp of requirements until we've used something for a while. It seems to be working well for us so far, but it's still early. There are lots of answers here that aren't helm, so I'm curious if…

Are you doing what Gitlab claims below "We (GitLab) use it mostly as a templating system as well" or really using it to manage complex apps? I think it doesn't really offer much for normal microservices.

I do use it for things like nginx ingress but for stuff I've built a service.yml / deployment.yml are fine.

Re: Kubernetes at GitHub

#36

Post by rleigh »

AaronBBrown wrote: post #24
cyphar wrote: post #17

Earlier quoted context omitted.

I would be interested to know what storage driver they're using for their nodes. High container churn puts a lot of stress on the VFS subsystem of Linux, and we've seen cases where customers have trigger lots of mount/umounts which results in filesystems causing panics. At SUSE, we do have some kernel devs debugging the issues, but the workaround is almost always "rate limit all the things". There are a few other ker…

Hey, this is Aaron from GitHub. We're using devicemapper w/ LVM backed pools. Would love to hear about your experience there. We definitely see this problem during periods of high container churn.

My schroot tool used for building Debian packages could panic a kernel in under five minutes reliably, when it was rapidly creating and destroying LVM snapshots in parallel (24 parallel jobs, with lifetimes ranging from seconds to hours, median a minute or so).

This was due to udev races in part (it likes to open and poke around with LVs in response to a trigger on creation, which races with deletion if it's very quick). I've seen undeletable LVs and snapshots, oopses and full lockups of the kernel with no panic. This stuff appears not to have been stress tested.

I switched to Btrfs snapshots which were more reliable but the rapid snapshot churn would unbalance it to read only state in just 18 hours or so. Overlays worked but with caveats. We ended up going back to unpacking tarballs for reliability. Currently writing ZFS snapshot support; should have done it years ago instead of bothering with Btrfs.

Re: Kubernetes at GitHub

#37

Post by yebyen »

awill wrote: post #21
deadmik3 wrote: post #7

Earlier quoted context omitted.

I'd recommend looking into openshift. it's basically kubectl + cool deployment features. there's also free, paid, and dedicated online hosted options. disclaimer: I work on openshift

You had a big announcement about openshift.io. Everyone on HN signed up, but it's been months, but I'm still 'awaiting approval' Have you let anyone in? What's the value in all the marketing hype if you then don't let people in.

Seconded. The last e-mail I got from OpenShift.io indicated they haven't let anyone in. I've already practically lost interest. It was evidently only "coming soon" but that announcement really looked like "coming tomorrow."

The original OpenShift Developer Preview made you sign up, but you would be allowed into the platform within hours or days.

Re: Kubernetes at GitHub

#38

Post by howinator »

Yeroc wrote: post #9

One thing I would have liked to have seen addressed in the article is whether the new architecture requires additional hardware (presumably) to operate and if so how much more.

We're on AWS, so this is tangentially related. At my company, we moved to k8s because we have quite a few low-usage services. Before k8s, each one of those services was getting its own EC2 instance. After k8s, we just have one set of machines which all the services use. If one service is getting more traffic, the resources for that service scale up, but we maintain a low baseline resource usage. In short, it's resulted in a measurable drop in our EC2 usage.

Re: Kubernetes at GitHub

#39

Post by alpb »

sandGorgon wrote: post #33

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

Can you elaborate? What are the issues you're seeing with Ingress?

Re: Kubernetes at GitHub

#40

Post by henridf »

bryanlarsen wrote: post #10
DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

We're using helm, but that was chosen mostly based on gut feel. It's an official project, and has momentum. We didn't want to spend too much time choosing a tool until we knew our requirements, and we don't really have a firm grasp of requirements until we've used something for a while. It seems to be working well for us so far, but it's still early. There are lots of answers here that aren't helm, so I'm curious if…

We ran into a number of issues with Helm when deploying - failures leading us to have to rollback, with rollbacks then failing, requiring manual changes to unblock.

I think that for third-party packages and related templating (which seems like the original use-case) it works well, but I would be wary of using it for high-res deploys of our own stuff.

Post reply on HN