Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes at GitHub

githubengineering.com

Post reply on HN
Print view View original

81–90 of 142 posts

Re: Kubernetes at GitHub

#81

Post by sandGorgon »

brown9-2 wrote: post #60
sandGorgon wrote: post #33

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

What does Swarm do differently here?

swarm is a batteries-included system. you can use it in the way that kubernetes is used... or (if you dont have all these sophisticated load balancers), you can allow it to load balance for you.

https://docs.docker.com/engine/swarm/ingress/#publish-a-port...

What it means is that when you create a docker swarm - it starts working.

Re: Kubernetes at GitHub

#82

Post by theptip »

DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

A lot of folks are using Helm, but I find it very opaque to debug when templates go wrong (and I feel quite strongly that we shouldn't be writing untyped templates for our models). Also I found writing reusable spec components to be very difficult, e.g. a reverse proxy that I add to a number of pods.

I use pykube (also worth looking at the incubator project client-python) to write deploy scripts in Python; client-python is particularly nice as it uses OpenAPI specs to give you type hinting on the API objects/fields that you're writing. Much more civilized than typing bare yaml into a text editor.

If Python isn't your thing you can generate your own client from the OpenAPI specs, though I've found the client generation process to be a bit buggy.

Re: Kubernetes at GitHub

#84

Post by EtienneK »

lima wrote: post #75
iagooar wrote: post #46

Love to see more Kubernetes success stories. I work for an ISP and we are trying to write another success story ;) As an ISP, we have tons of constraints in terms of infrastructure. We're not allowed to use any public cloud services. At the same time, the in-house infrastructure is either too limited, or managed via spreadsheets by a bunch of dysfunctional teams. For my team, Kubernetes has been truly a life saver wh…

Red Hat's OpenShift makes it a lot easier by providing all of the infrastructure around it (docker registry, docker build from Git, Ansible integration and so on). Best docs of all open source projects I've seen.

I second this. Have been PoCing OpenShift for a couple of months now and it's been a joy to use.

Re: Kubernetes at GitHub

#85

Post by rubenv »

DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

I recently spoke about the approach we use at Ticketmatic: https://rocketeer.be/articles/coreos-fest-2017/

Re: Kubernetes at GitHub

#86

Post by ejholmes »

daxfohl wrote: post #83

Curious, being a RoR app, did github ever run on Heroku? (Obviously googling "github heroku" is just a million tutorials on how to integrate.)

I think I remember hearing that they were on EngineYard back in the day, but never Heroku.

Re: Kubernetes at GitHub

#87

Post by thockingoog »

sandGorgon wrote: post #33

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

This is an apt point. Kubernetes models Borg, and Borg has no concept of ingress. That's an entirely different problem space.

Obviously that doesn't fly if there isn't an equivalent open solution, so we did what we could with the system to make it not terrible. We can do more.

The point about Swarm is interesting, and has been much on my mind. Some of Kubernetes' perceived complexity is because we go to great lengths to avoid ever having two users collide, with escape hatches for the people who really need "unfriendly" features. This is because, again, Kubernetes models Borg. Borg clusters are giant, shared, multi-user, multi-app animals, where the users are in different business units and chances of collisions are high.

Swarm, on the other hand, thinks of a cluster more as an application construct. Sharing is not a big problem, and coordination is easy and local. This allows them to make different tradeoffs. I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports.

I still believe the large-shared-cluster model is right in the limit. There are so many efficiencies to be had. But there are legit reasons it is hard to achieve right now.

I'm very interested in ways to make Kubernetes easier to use, ESPECIALLY in this regard. Real user feedback is critical.

Re: Kubernetes at GitHub

#88

Post by kalev »

rubenv wrote: post #85
DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

I recently spoke about the approach we use at Ticketmatic: https://rocketeer.be/articles/coreos-fest-2017/

That must have been an awesome talk. Thanks for the write-up!

Re: Kubernetes at GitHub

#89

Post by sandGorgon »

thockingoog wrote: post #87
sandGorgon wrote: post #33

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

This is an apt point. Kubernetes models Borg, and Borg has no concept of ingress. That's an entirely different problem space. Obviously that doesn't fly if there isn't an equivalent open solution, so we did what we could with the system to make it not terrible. We can do more. The point about Swarm is interesting, and has been much on my mind. Some of Kubernetes' perceived complexity is because we go to great lengths…

that's an interesting perspective - however, Docker Swarm also does that. Docker Swarm secrets have been GA for longer thank k8s. The new UCP mechanism in the Docker Datacenter product is fairly interesting (has not made it to swarm yet). Its a paid product but makes RBAC pretty simple - https://success.docker.com/Architecture/Docker_Reference_Arc... .

One very interesting tool that Docker makes available is https://store.docker.com/community/images/docker/docker-benc...

I think the issue with k8s is that it is competing with the "Ruby on Rails" of frameworks viz Docker Swarm. I think the pluggability of critical pieces like ingress and secrets was taken too far.

> I doubt very much that you can run a large number of similar apps in a single swarm without having collisions on things like ports.

I dont think that is true, it does manage its overlay networks pretty well. Which is FWIW, another place where k8s took the non-opinionatedness too far. I think the number of bugs on "my stuff doesnt work with flannel but works with calico" should tell you that.

To be honest, Docker Swarm has some of these issues as well - https://github.com/moby/moby/issues/25526. But the fixes are included in the "batteries". On kuberenetes, I have to run behind upstream projects with heterogenous configuration (nginx vs haproxy ingress. or flannel vs calico configuration) to try and fix it.

Re: Kubernetes at GitHub

#90

Post by sweettea »

rleigh wrote: post #36
AaronBBrown wrote: post #24

Earlier quoted context omitted.

Hey, this is Aaron from GitHub. We're using devicemapper w/ LVM backed pools. Would love to hear about your experience there. We definitely see this problem during periods of high container churn.

My schroot tool used for building Debian packages could panic a kernel in under five minutes reliably, when it was rapidly creating and destroying LVM snapshots in parallel (24 parallel jobs, with lifetimes ranging from seconds to hours, median a minute or so). This was due to udev races in part (it likes to open and poke around with LVs in response to a trigger on creation, which races with deletion if it's very qui…

In my work identity, we saw a similar problem in our testing, where blkid would cause undesired IO on fresh devices. Eventually, we disabled blkid scanning our device mapper devices upon state changes with a file /etc/udev/59-no-scanning-our-devices.rules containing: ENV{DM_NAME}=="ourdevice", OPTIONS:="nowatch"

Alternately, you could call 'udevadm settle' after device creation before doing anything else, which will let blkid get its desired IO done, I think.

Post reply on HN