Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let’s Encrypt, OAuth 2, and Kubernetes Ingress

eng.fromatob.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#11

Post by morgante »

agentgt wrote: post #8
fortytw2 wrote: post #7

Earlier quoted context omitted.

For the moment at least, it's much more comfortable for us to keep our databases outside of kubernetes, so we use saltstack(masterless), packer, and terraform to manage them. We also use terraform to manage all of our DNS, which is split between Route53 and the GCP DNS service.

Thanks! I have been meaning to give terraform a try to replace some of our custom gcloud + gcloud deployment descriptors. Also so that we don't need a separate docker compose version for development (I'm assuming in theory you can run terraform to do what docker compose does?).

> Also so that we don't need a separate docker compose version for development (I'm assuming in theory you can run terraform to do what docker compose does?).

Not really. Terraform is much more meant as a tool for manipulating production infrastructure (primarily clouds), not for orchestrating Docker containers (including locally).

I'd strongly recommend you use the right tool for the job and it's a very rare job where Terraform is a good alternative to docker-compose.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#12

Post by fortytw2 »

andrewstuart2 wrote: post #9

Suggestion to anybody reading this: don't use a DaemonSet for this. This really ought to be a Deployment of nginx-ingress resources behind a service exposed as `type: LoadBalancer` (if you're in a cloud-provider that supports LoadBalancer services). Then just create DNS aliases and configure nginx to do session affinity if needed, etc. Not only will it be able to scale with your load instead of cluster size, but you…

So I (the author) am a bit torn on this - a `type: LoadBalancer` service will create a NodePort underneath (yet another internal loadbalancer) and map those ports to a $cloud-platform-tcp-loadbalancer. By using a DaemonSet with a host port bound, you avoid a layer of internal routing.

I'm not so sure if one approach is _particularly_ better than the other though.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#13

Post by andrewstuart2 »

fortytw2 wrote: post #12
andrewstuart2 wrote: post #9

Suggestion to anybody reading this: don't use a DaemonSet for this. This really ought to be a Deployment of nginx-ingress resources behind a service exposed as `type: LoadBalancer` (if you're in a cloud-provider that supports LoadBalancer services). Then just create DNS aliases and configure nginx to do session affinity if needed, etc. Not only will it be able to scale with your load instead of cluster size, but you…

So I (the author) am a bit torn on this - a `type: LoadBalancer` service will create a NodePort underneath (yet another internal loadbalancer) and map those ports to a $cloud-platform-tcp-loadbalancer. By using a DaemonSet with a host port bound, you avoid a layer of internal routing. I'm not so sure if one approach is _particularly_ better than the other though.

As far as the routing layer, that's going to be a very small gain as it's all handled by the kernel via iptables anyway. What you give up by using a DaemonSet is a constant overhead for every single worker node that gets created. You have to set aside capacity for traffic you might not have.

A Deployment can be scaled by actual utilization of the pod, and in the near future (if not already) via custom metrics of your own design (e.g. prometheus).

So if you have to spin up 50 new nodes to handle some batch machine learning work, you're going to wastefully create 50 new nginx instances when there's no new ingress traffic to handle. With a deployment, it just scales naturally as needed. So it's not about the marginal gains, it's about using the right tool for the job. :-)

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#14

Post by fortytw2 »

andrewstuart2 wrote: post #13
fortytw2 wrote: post #12

Earlier quoted context omitted.

So I (the author) am a bit torn on this - a `type: LoadBalancer` service will create a NodePort underneath (yet another internal loadbalancer) and map those ports to a $cloud-platform-tcp-loadbalancer. By using a DaemonSet with a host port bound, you avoid a layer of internal routing. I'm not so sure if one approach is _particularly_ better than the other though.

As far as the routing layer, that's going to be a very small gain as it's all handled by the kernel via iptables anyway. What you give up by using a DaemonSet is a constant overhead for every single worker node that gets created. You have to set aside capacity for traffic you might not have. A Deployment can be scaled by actual utilization of the pod, and in the near future (if not already) via custom metrics of your…

Fair enough points :) I'll see about reworking the article, and our deployment a bit - keeping in mind that LoadBalancer services are platform dependent

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#15

Post by caleblloyd »

andrewstuart2 wrote: post #9

Suggestion to anybody reading this: don't use a DaemonSet for this. This really ought to be a Deployment of nginx-ingress resources behind a service exposed as `type: LoadBalancer` (if you're in a cloud-provider that supports LoadBalancer services). Then just create DNS aliases and configure nginx to do session affinity if needed, etc. Not only will it be able to scale with your load instead of cluster size, but you…

In order to get the proper Source IP with TCP Load Balancing before Kubernetes 1.5, you used to need to use a Daemon Set with Host Networking.

Kubernetes 1.5 introduced Source IP using Source NAT and Health Checks: https://kubernetes.io/docs/tutorials/services/source-ip/

You still must write scheduling rules so that pods are scheduled to have at most 1 instance running per node.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#17

Post by theptip »

Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted.

That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear.

There are a couple of pieces of hard-to-find config required to achieve TLS-to-the-pod with Ingress:

1) You need to enable ssl-passthrough on your nginx ingress; this is a simple annotation: https://github.com/kubernetes/contrib/issues/1854. This will use nginx's streaming mode to route requests with SNI without terminating the TLS connection.

2) Now you'll need a way of getting your certs into the pod; kube-lego attaches the certs to the Ingress pod, which is not what you want for TLS-to-the-pod. https://github.com/PalmStoneGames/kube-cert-manager/ lets you do this in an automated way, by creating k8s secrets containing the letsencrypt certs.

3) Your pods will need an SSL proxy to terminate the TLS connection. I use a modified version of https://github.com/GoogleCloudPlatform/nginx-ssl-proxy.

4) You'll want a way to dynamically create DNS entries; Mate is a good approach here. Note that once you enable automatic DNS names for your Services, then it becomes less important to share a single public IP using SNI. You can actually abandon the Ingress, and have Mate set up your generated DNS records to point to the Service's LoadBalancer IP.

(As an aside, if you stick with Nginx Ingress, you can connect it to the outside world using a Kubernetes loadbalancer, instead of having to use a Terraform LB; the (hard-to-find and fairly new) config flag for that is `publish-service` (https://github.com/kubernetes/ingress/blob/master/core/pkg/i...).

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#18

Post by conradev »

theptip wrote: post #17

Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted. That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear. There are a couple of pieces of hard-to-find config…

Would an overlay network with a shared secret for encryption of pod-to-pod networking be another solution to this problem? I feel like the ideal should involve keeping the key material in as few places as possible.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#19

Post by SEJeff »

theptip wrote: post #17

Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted. That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear. There are a couple of pieces of hard-to-find config…

Yes and no.

You're sending traffic over google's SDN in the clear, which is still encrypted by google if you believe:

https://cloud.google.com/security/security-design/

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#20

Post by theptip »

SEJeff wrote: post #19
theptip wrote: post #17

Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted. That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear. There are a couple of pieces of hard-to-find config…

Yes and no. You're sending traffic over google's SDN in the clear, which is still encrypted by google if you believe: https://cloud.google.com/security/security-design/

I am fairly sure that traffic between nodes is not encrypted; I have dug into this in some detail with GCP staff. if you can quote some sources to the contrary I'd be interested to see them.

(The linked security design doc mentions 'encrypted in transit _to the data center_', but it doesn't address traffic inside the DC, last time I read through it in detail).

Post reply on HN