Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes at GitHub

githubengineering.com

Post reply on HN
Print view View original

51–60 of 142 posts

Re: Kubernetes at GitHub

#51

Post by howinator »

foxylion wrote: post #6
DDub wrote: post #2

We're currently looking at moving our applications to k8s, and was wondering what deployment tools people are using? This week we are evaluating spinnaker, helm and bash wrappers for kubectl. There is concern over adding too many layers of abstraction and that KISS is the best approach.

We also did some evaluation and then decided to stick to KISS an chose kubectl commands combined with cat and kexpand. Really simple approach to allow dynamic kubernetes deployments. Example command can be cat service.yml | kexpand expand -v image-tag=git-135afed4 | kubectl apply -f - The service.yml contains the full deployment configuration, service definition and ingress rules. So this works without preconfiguring…

We basically did the same thing except we used Ansible for our templating. This allows us to store all our shared "environment configuration data", e.g., name of RDS for services in prod environment, name of backup S3 bucket for services in dev environment, in an Ansible role then just pull that information into our templated deployment manifest file. So far, it's worked out pretty well for us.

Re: Kubernetes at GitHub

#52

Post by ecnahc515 »

lobster_johnson wrote: post #25
sytse wrote: post #22

Earlier quoted context omitted.

At GitLab we recommend to use CPU cores + 1 as the number of unicorn workers https://docs.gitlab.com/ce/install/requirements.html#unicorn...

How do you configure that? A pod doesn't know what machine it's running on ahead of time. You can create nodepools and use node selectors to pin the pod to that nodepool, but I'm not sure I love the idea.

Using the downward API, your application can get the number of CPU cores or the millicore value via environment variables or volumes. This would let you configure your number of workers based on what resource limits you set on your container.

Re: Kubernetes at GitHub

#53

Post by skewart »

> Several qualities of Kubernetes stood out from the other platforms we evaluated: the vibrant open source community supporting the project, the first run experience (which allowed us to deploy a small cluster and an application in the first few hours of our initial experiment), and a wealth of information available about the experience that motivated its design.

It's interesting that the reasons they cite for choosing Kubernetes over alternatives are entirely driven by 'developer experience' and not at all technical. It shows how critical community development, good documentation, and marketing are to building a successful open source project.

Re: Kubernetes at GitHub

#54

Post by robotmay »

philips wrote: post #42

Really exciting stuff, happy to see the Github team launch this. Kubernetes is becoming the goto for folks needing both their own physical metal presence and cloud footprint too. And the magic of Kubernetes is that it has APIs that can actually give teams the confidence to run and reuse deployment strategies in all environments. Even across clouds. If you are like Github and want to use Kubernetes across clouds (AWS,…

I'm still utterly perplexed as to what Tectonic actually -is-. I kinda get that it's a kubernetes setup, but is it a GUI over the top of it? The website is pretty confusing and I think I gave up really quickly when trying to set it up.

Re: Kubernetes at GitHub

#55

Post by mnutt »

alpb wrote: post #39
sandGorgon wrote: post #33

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

Can you elaborate? What are the issues you're seeing with Ingress?

The service I'm working on porting to kubernetes has bandwidth requirements that far exceed a single machine, and though we're on AWS, we can't use ELB for various reasons. We ended up with a ReplicaSet of haproxies using HostPort, and a separate app that watches for haproxy service changes and updates Route53 round-robin DNS. We're somewhat fortunate that we only have a single service that needs to use the http/https HostPort.

We could have just left our haproxies outside of kubernetes, and may eventually end up doing so if the network performance doesn't meet our needs. As it is, it all works but there are a ton of sidecar services all over the place.

Re: Kubernetes at GitHub

#56

Post by brianwawok »

nikon wrote: post #31
foxylion wrote: post #6

Earlier quoted context omitted.

We also did some evaluation and then decided to stick to KISS an chose kubectl commands combined with cat and kexpand. Really simple approach to allow dynamic kubernetes deployments. Example command can be cat service.yml | kexpand expand -v image-tag=git-135afed4 | kubectl apply -f - The service.yml contains the full deployment configuration, service definition and ingress rules. So this works without preconfiguring…

I do something similar, but envsubst does the job.

Ditto me too to get my git sha into my image names and deploy with deployments.

Re: Kubernetes at GitHub

#57

Post by spmurrayzzz »

Alan01252 wrote: post #44

I'm curious to what this means for the existing puppet code base, is it now irrelevant, or are there still usages for it in the k8s world?

They could easily still use standalone puppet to handle the config management for individual container images. I currently do this with salt-minion. It reduces the burden on the Dockerfile itself, and lets you embrace a declarative configuration state at build time.

Re: Kubernetes at GitHub

#58

Post by yebyen »

mnutt wrote: post #55
alpb wrote: post #39

Earlier quoted context omitted.

Can you elaborate? What are the issues you're seeing with Ingress?

The service I'm working on porting to kubernetes has bandwidth requirements that far exceed a single machine, and though we're on AWS, we can't use ELB for various reasons. We ended up with a ReplicaSet of haproxies using HostPort, and a separate app that watches for haproxy service changes and updates Route53 round-robin DNS. We're somewhat fortunate that we only have a single service that needs to use the http/http…

This sounds like what Deis Router was created to do.

It sounds like you've already got it nailed down, but maybe like to have a look at this: https://github.com/deis/router

(It's probably tightly coupled to Deis Controller, but something to look at anyway!)

Re: Kubernetes at GitHub

#59

Post by philips »

robotmay wrote: post #54
philips wrote: post #42

Really exciting stuff, happy to see the Github team launch this. Kubernetes is becoming the goto for folks needing both their own physical metal presence and cloud footprint too. And the magic of Kubernetes is that it has APIs that can actually give teams the confidence to run and reuse deployment strategies in all environments. Even across clouds. If you are like Github and want to use Kubernetes across clouds (AWS,…

I'm still utterly perplexed as to what Tectonic actually -is-. I kinda get that it's a kubernetes setup, but is it a GUI over the top of it? The website is pretty confusing and I think I gave up really quickly when trying to set it up.

Tectonic is Enterprise Kubernetes. We start with pure upstream Kubernetes at the core and install it in a production ready setup with the Tectonic Installer[1] across clouds or bare metal. On top of those basics Tectonic provides things most organizations need:

- Authentication backed by LDAP/SAML/etc

- One-click automated updates of the entire cluster

- Pre-configured cluster monitoring/alerting

There is a bunch more in there and in the roadmap too but that gives you a taste.

The other thing is that we provide professional services, training, and support to customers on the whole stack from the VM or machine on up to the Kubernetes API. We have done neat collaborations with customers like the ALB Ingress Controller[2] too.

[1] https://github.com/coreos/tectonic-installer [2] https://github.com/coreos/alb-ingress-controller

Re: Kubernetes at GitHub

#60

Post by brown9-2 »

sandGorgon wrote: post #33

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

What does Swarm do differently here?
Post reply on HN