Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubernetes at GitHub

githubengineering.com

Post reply on HN
Print view View original

71–80 of 142 posts

Re: Kubernetes at GitHub

#71

Post by twk3 »

lobster_johnson wrote: post #25
sytse wrote: post #22

Earlier quoted context omitted.

At GitLab we recommend to use CPU cores + 1 as the number of unicorn workers https://docs.gitlab.com/ce/install/requirements.html#unicorn...

How do you configure that? A pod doesn't know what machine it's running on ahead of time. You can create nodepools and use node selectors to pin the pod to that nodepool, but I'm not sure I love the idea.

Our entrypoint configures the unicorn workers before starting using a chef omnibus call. So we grab the memory and cpus using Ohai: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/fil... and do cpus+1

^This is pretty much we do by default for all our regular package installs, but in some of our Kubernetes Helm charts we instead statically configure the pod resources and unicorn workers. (Defaulting to 1cpu - 2workers per front-end pod) eg: https://gitlab.com/charts/charts.gitlab.io/blob/master/chart...

As someone mentioned in this thread, using the downward api might be a cool way to configure the workers.

Re: Kubernetes at GitHub

#72

Post by codesuki »

kelseyhightower wrote: post #45
sandGorgon wrote: post #33

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

Any load balancer can be configured or modified to target routable Pod IP addresses and skip node ports altogether. You'll have to integrate with the Kubernetes Endpoints API[1] and support dynamic backends. Another option would be to leverage Kubernetes' DNS and the SRV records[2] backing each service. The reason node ports are used in the Cloud today is because most Cloud load balancing solutions only target VMs, n…

Sorry to turn this into a different direction but I wanted to ask you about packaging + deployment.

I am about to try using helm for packaging my Kubernetes configs to make use of its templating. Being able to include Kubernetes changes in the release makes it less common to forget some new env variable etc.

The only thing I don't like is that it replaces kubectl in a way. And some comments here speak of problems during deploy and rollback which makes me wary.

What are some best practices around that topic? I was also thinking of deploying with spinnaker but could not confirm it works with helm.

Any info is appreciated!

P.S. have you thought about making a redis-cluster example repo? Redis 4.0 has a new `cluster_announce_ip` setting with which I made it work, but I still don't like my setup 100%.

Re: Kubernetes at GitHub

#73

Post by TheDong »

joombaga wrote: post #70
erulabs wrote: post #67

If you're running Kube on AWS, make sure you install the proper drivers! For Ubuntu, that's the `linux-aws` apt package. https://github.com/kubernetes/kops/issues/1558 Missing ENA and ixgbevf can be a real performance killer!

Is this used by vanilla docker / ECS, or just k8s?

that advice holds regardless what software you're using, so long as the software does network traffic.

It holds for just running plain old nginx websites.

It doesn't really matter for small instance sizes where your networking is already rate-limited by amazon so much that ENA drivers won't matter, but on beefy instances it's always good advice to make sure you're using ENA supported driverse.

See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced...

Re: Kubernetes at GitHub

#74

Post by bmoyles »

erulabs wrote: post #67

If you're running Kube on AWS, make sure you install the proper drivers! For Ubuntu, that's the `linux-aws` apt package. https://github.com/kubernetes/kops/issues/1558 Missing ENA and ixgbevf can be a real performance killer!

FWIW, the stock kernel (and HWE/HWE-edge kernels) recently picked up current ENA drivers. ixgbevf, unfortunately, doesn't look like it's been updated in-tree so it still lags behind Amazon's recommendation (currently 2.14.2, whereas Xenial's in-tree driver claims to be 2.12.1 and Trusty has 2.11.3).

Re: Kubernetes at GitHub

#75

Post by lima »

iagooar wrote: post #46

Love to see more Kubernetes success stories. I work for an ISP and we are trying to write another success story ;) As an ISP, we have tons of constraints in terms of infrastructure. We're not allowed to use any public cloud services. At the same time, the in-house infrastructure is either too limited, or managed via spreadsheets by a bunch of dysfunctional teams. For my team, Kubernetes has been truly a life saver wh…

Red Hat's OpenShift makes it a lot easier by providing all of the infrastructure around it (docker registry, docker build from Git, Ansible integration and so on).

Best docs of all open source projects I've seen.

Re: Kubernetes at GitHub

#77

Post by AlexB138 »

drdaeman wrote: post #69

How hard (and how realistic) it is to actually get a reasonable understanding (and then stay up-to-date) with Kubernetes internals? Is there any go-to reading material? We had ran another large-footprint container management system (not K8s, but also popular), and when its DNS component started to eat all the CPU on all nodes, best I was able to do fast,was just scrapping the whole thing and quickly replacing it with…

I wouldn't say that you're unnecessarily scaring yourself at all. Kubernetes is extremely complex. I've been running it for a few months and I'm just starting to get my hands around it. Things will just stop working for what seems like no reason, and there are so many places to investigate you can easily burn most of a day troubleshooting.

It's a great system, but it's also relatively new, and most issues aren't well documented. You'll spend a lot of time in github issues or asking for help in the (very active, and often very helpful) community.

If you have a valid use case, I wouldn't steer you away from it, but your fears are well founded.

Re: Kubernetes at GitHub

#78

Post by cpitman »

awill wrote: post #21
deadmik3 wrote: post #7

Earlier quoted context omitted.

I'd recommend looking into openshift. it's basically kubectl + cool deployment features. there's also free, paid, and dedicated online hosted options. disclaimer: I work on openshift

You had a big announcement about openshift.io. Everyone on HN signed up, but it's been months, but I'm still 'awaiting approval' Have you let anyone in? What's the value in all the marketing hype if you then don't let people in.

Remember that OpenShift.io is just one (newer) way to work with OpenShift, currently focused on deploying to the hosted OpenShift Online.

But if you are looking at doing self-hosted Kubernetes, you really should give OpenShift a look. It does a lot to give sane defaults for things like ingress and building containers, while simplifying the deployment story.

For just getting started and kicking the tires, you can use minishift to spin up a local VM with everything installed and configured: https://www.openshift.org/minishift/

Disclaimer: I also work for Red Hat. We're everywhere :-)

Re: Kubernetes at GitHub

#79

Post by [deleted] »

codesuki wrote: post #72
kelseyhightower wrote: post #45

Earlier quoted context omitted.

Any load balancer can be configured or modified to target routable Pod IP addresses and skip node ports altogether. You'll have to integrate with the Kubernetes Endpoints API[1] and support dynamic backends. Another option would be to leverage Kubernetes' DNS and the SRV records[2] backing each service. The reason node ports are used in the Cloud today is because most Cloud load balancing solutions only target VMs, n…

Sorry to turn this into a different direction but I wanted to ask you about packaging + deployment. I am about to try using helm for packaging my Kubernetes configs to make use of its templating. Being able to include Kubernetes changes in the release makes it less common to forget some new env variable etc. The only thing I don't like is that it replaces kubectl in a way. And some comments here speak of problems dur…

[deleted]

Re: Kubernetes at GitHub

#80

Post by sandGorgon »

kelseyhightower wrote: post #45
sandGorgon wrote: post #33

> We enhanced GLB, our internal load balancing service, to support Kubernetes NodePort Services. Everyone does this - because Kubernetes Achilles heel is its ingress. It is still built philosophically as a post-loadbalancing system . This is the single biggest reason why using Docker Swarm is so pleasant.

Any load balancer can be configured or modified to target routable Pod IP addresses and skip node ports altogether. You'll have to integrate with the Kubernetes Endpoints API[1] and support dynamic backends. Another option would be to leverage Kubernetes' DNS and the SRV records[2] backing each service. The reason node ports are used in the Cloud today is because most Cloud load balancing solutions only target VMs, n…

Hi Kelsey, thanks for replying. You are definitely k8s secret weapon ;)

I know about this. However, ultimately the question is that Kubernetes is not a gradual scale up solution for most people. I have to be prepared to deal with building my own load balancer.

Basically, I cannot do an on-metal deployment very easily. Most of the questions on k8s slack for metal deployments were - how do I set this up with few tweaks like ssl pass through and source ip preservation.

It is not easy.

Either you build your own load balancer or you use a cloud provided one. Now, Ingresses are not pleasant. I'm not sure about the state of source ip preservation, but last I remember that the nginx ingress had still not surfaced ssl_preread_server_name to the ingress configuration.

Now, what would have been nice is if it was ingress-all-the-way-down : ingress with something like istio/linkerd, maybe it is possible.

Tl;Dr - I'm not github. I can't build my own load balancer. Give me something that works out of the box. Yes, I know it may go down - I'll survive. Docker Swarm does this.

Post reply on HN