Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let’s Encrypt, OAuth 2, and Kubernetes Ingress

eng.fromatob.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#21

Post by theptip »

conradev wrote: post #18
theptip wrote: post #17

Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted. That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear. There are a couple of pieces of hard-to-find config…

Would an overlay network with a shared secret for encryption of pod-to-pod networking be another solution to this problem? I feel like the ideal should involve keeping the key material in as few places as possible.

Yes, that's an option -- the simplest option would be a single overlay network, with the node-node tunnels encrypted using IPSec or similar (https://github.com/coreos/flannel/issues/6 or https://www.weave.works/documentation/net-latest-how-it-work... or https://github.com/projectcalico/felix/issues/997). I think this would be tricky to configure in GKE though.

With a secret-per-pod, your key material lives in in the etcd on the API server, and gets mounted in a tmpfs on each pod that is given the secret. Only Pods in the Secret's namespace can access the Secret, so if you have RBAC configured correctly it should be possible to lock this down tightly to only the code that needs the Secret. (I'm not sure how to do this in GKE; I'm currently treating each cluster as a single security domain).

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#22

Post by A-F1V3 »

theptip wrote: post #21
conradev wrote: post #18

Earlier quoted context omitted.

Would an overlay network with a shared secret for encryption of pod-to-pod networking be another solution to this problem? I feel like the ideal should involve keeping the key material in as few places as possible.

Yes, that's an option -- the simplest option would be a single overlay network, with the node-node tunnels encrypted using IPSec or similar ( https://github.com/coreos/flannel/issues/6 or https://www.weave.works/documentation/net-latest-how-it-work... or https://github.com/projectcalico/felix/issues/997 ). I think this would be tricky to configure in GKE though. With a secret-per-pod, your key material lives in in th…

If you are open to using an (awesome) vendor for this kind of thing, I would highly recommend taking a look at Backplane (https://www.backplane.io/). It's as simple as running a sidecar Backplane agent container along side any container that you want to route http traffic to and then shape the traffic via their API (it becomes your load balancer as well). They automatically provision Lets Encrypt certificates for your endpoints as well, so you don't have to worry about any of that. We have been using it at Mux in production for months and have been very happy with the results. Backplane also has some other nice built in features like blue/green deploys and built in OAuth support that are really nice to have out of the box.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#23

Post by captn3m0 »

>On GCP, the HTTP load balancers do not support TLS-SNI, which means you need a new frontend IP address per SSL certificate you have. For internal services, this is a pain, as you cannot point a wildcard DNS entry to a single IP, like * .fromatob.com and then have everything just work.

Wouldn't a wildcard SSL cert + wildcard DNS entry work even without SNI support here? I haven't used the GCP load balancer, but as long as you are serving a single certificate (* .fromatob.com), the client/server don't have to rely on SNI at all.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#24

Post by lobster_johnson »

theptip wrote: post #17

Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted. That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear. There are a couple of pieces of hard-to-find config…

I wonder how much of a vulnerability that really is. The SDN encapsulates everything and is supposedly IP-spoofing-secure, so in principle there's no way for anyone else in the same DC to get your traffic.

Of course, you could have a local attacker get in through other means, and then access local DC traffic within your SDN. But if you get to that point, you probably have bigger problem than terminating SSL.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#25

Post by SEJeff »

theptip wrote: post #20
SEJeff wrote: post #19

Earlier quoted context omitted.

Yes and no. You're sending traffic over google's SDN in the clear, which is still encrypted by google if you believe: https://cloud.google.com/security/security-design/

I am fairly sure that traffic between nodes is not encrypted; I have dug into this in some detail with GCP staff. if you can quote some sources to the contrary I'd be interested to see them. (The linked security design doc mentions 'encrypted in transit _to the data center_', but it doesn't address traffic inside the DC, last time I read through it in detail).

No I suspect you're right even though the linked security design doc does say they encapsulate all app level traffic (such as HTTP) in their own RPC.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#26

Post by theptip »

lobster_johnson wrote: post #24
theptip wrote: post #17

Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted. That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear. There are a couple of pieces of hard-to-find config…

I wonder how much of a vulnerability that really is. The SDN encapsulates everything and is supposedly IP-spoofing-secure, so in principle there's no way for anyone else in the same DC to get your traffic. Of course, you could have a local attacker get in through other means, and then access local DC traffic within your SDN. But if you get to that point, you probably have bigger problem than terminating SSL.

Your main attack vectors are:

1) A disgruntled employee sets up a surreptitious tap on the network to see if any secret material comes through. A high value target would be `Authorization: Bearer` in HTTP headers, but there are plenty of other things to slurp up.

2) A normally honest employee running an unrelated network tap to diagnose an issue with the SDN spots your Authorization headers (or other secrets), and knowing that they have a legitimate reason to have the wire capture, copies out the key material. This is very hard to prevent, since network admins can and should be tapping the network from time to time.

I'm not particularly concerned about someone from outside Google breaking into the SDN fabric, though a hypervisor breach could leak network traffic from other tenants on your instance (if you are sharing).

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#27

Post by xkarga00 »

andrewstuart2 wrote: post #13
fortytw2 wrote: post #12

Earlier quoted context omitted.

So I (the author) am a bit torn on this - a `type: LoadBalancer` service will create a NodePort underneath (yet another internal loadbalancer) and map those ports to a $cloud-platform-tcp-loadbalancer. By using a DaemonSet with a host port bound, you avoid a layer of internal routing. I'm not so sure if one approach is _particularly_ better than the other though.

As far as the routing layer, that's going to be a very small gain as it's all handled by the kernel via iptables anyway. What you give up by using a DaemonSet is a constant overhead for every single worker node that gets created. You have to set aside capacity for traffic you might not have. A Deployment can be scaled by actual utilization of the pod, and in the near future (if not already) via custom metrics of your…

FWIW, you can constrain a DaemonSet to run on a specific set of nodes by using nodeAffinity but you would need to label those nodes appropriately which you are going to do anyway in a production cluster (use a specific set of nodes for running infra components such as routers or registries).

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#28

Post by mrkurt »

andrewstuart2 wrote: post #13
fortytw2 wrote: post #12

Earlier quoted context omitted.

So I (the author) am a bit torn on this - a `type: LoadBalancer` service will create a NodePort underneath (yet another internal loadbalancer) and map those ports to a $cloud-platform-tcp-loadbalancer. By using a DaemonSet with a host port bound, you avoid a layer of internal routing. I'm not so sure if one approach is _particularly_ better than the other though.

As far as the routing layer, that's going to be a very small gain as it's all handled by the kernel via iptables anyway. What you give up by using a DaemonSet is a constant overhead for every single worker node that gets created. You have to set aside capacity for traffic you might not have. A Deployment can be scaled by actual utilization of the pod, and in the near future (if not already) via custom metrics of your…

Services are all handled with IP Tables, but in current versions of kubernetes you end up sending a lot of traffic to different nodes for no reason. There's an issue to prevent that, and send traffic to the pod on the local node if possible, but it's pretty gross right now.

This sucks for performance/reliability reasons. It also makes it crazy difficult to keep track of visitors' source IPs.

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#29

Post by prydonius »

This is a really great post, excited to give the OAuth 2 auth a try.

FWIW, an easier way to get started with the NGINX Ingress and kube-lego services is using the official Helm[1] Charts for them (https://github.com/kubernetes/charts/tree/master/stable/ngin... and https://github.com/kubernetes/charts/tree/master/stable/kube...).

[1] https://github.com/kubernetes/helm

Re: Let’s Encrypt, OAuth 2, and Kubernetes Ingress

#30

Post by maktouch »

andrewstuart2 wrote: post #9

Suggestion to anybody reading this: don't use a DaemonSet for this. This really ought to be a Deployment of nginx-ingress resources behind a service exposed as `type: LoadBalancer` (if you're in a cloud-provider that supports LoadBalancer services). Then just create DNS aliases and configure nginx to do session affinity if needed, etc. Not only will it be able to scale with your load instead of cluster size, but you…

We started nginx-ingress as a deployment, and we converted it to a DaemonSet:

- We rarely deploy new versions of the ingress controller

- We can't (or don't know how to) choose in which nodes the pods will go. If I make a deployment with 10 replicas, there's a chance that it'll all go in the same node

- Because we can't choose to distribute the pods, when a node containing the ingress pod went down, there was a noticeable blip of downtime (~27 seconds approx.). That's kinda unacceptable.

- Nginx ingress is pretty light. I don't mind it having just 1 of them in each node.

- Since we put our databases and stateful stuff outside kubernetes, we also decided to separate web facing kubernetes cluster and worker ones. This solves the problem of the "spin up 50 new nodes to handle some batch machine learning job".

So far, so good. I would actually suggest that you use a DaemonSet for this, just like I suggest you convert Kube-DNS to a daemonset (it's not by default on GKE for some obscure reason).

Post reply on HN