Kubernetes ingress-nginx controller vulnerability

For whatever it’s worth my startup began life as a multi tenant kubernetes provider. We have a ton of custom controls - and for example do not allow this “custom snippet” directive at all. We spent a very long time locking down as much of the api surface as possible.

I have war stories up the wazoo - at one point we had 16k random internet users on a single cluster!

>I can copy it, plug it into my browser, and inspect with ease the shell script that will run

If the source is some shady site, it's possible to serve two different versions: one for curl/wget users and one for browsers.

> "... a user who can create or update ingress objects, can use the custom snippets feature to obtain all secrets in the cluster."

To this point, handling of secrets in the Kubernetes ecosystem always seemed like a mess to me. Both `vault` and AWS SecretsManager/KMS in combination with IAM allow me to finely partition access to secrets by instance or developer identity whereas in the Kubernetes world one expects "secrets" to be handled by a small core group and machines seem to have mostly unfettered access.

Specifically[1],

>> Caution:

>> Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read any Secret in that namespace; this includes indirect access such as the ability to create a Deployment.

>> In order to safely use Secrets, take at least the following steps:

>> * Enable Encryption at Rest for Secrets[2].

>> * Enable or configure RBAC rules that restrict reading data in Secrets (including via indirect means).[3]

>> * Where appropriate, also use mechanisms such as RBAC to limit which principals are allowed to create new Secrets or replace existing ones.

When you dig deeper, both [2] and [3] look like they are someone's job security plan. No wonder it was developed at a place which employs people just to keep them off the market.

Especially since you really don't gain much unless you also have a KMS Provider[4].

No wonder most K8 setups I've seen still have BASE64 encoded secrets in some YAML file.

There is also no shortage of tutorial-spam -- so unless you happen to be lucky enough to apprentice with people who know what they are doing, it is hard to gain a solid understanding and move confidently.

[1]: https://kubernetes.io/docs/concepts/configuration/secret/

[2]: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-...

[3]: https://kubernetes.io/docs/reference/access-authn-authz/auth...

[4]: https://kubernetes.io/docs/tasks/administer-cluster/kms-prov...

> The only legitimate risk I've heard about 'curl | bash' is that web servers can detect whether the site is being accessed by a browser versus curl, and serve different content

This is the reason why you shouldn't do curl ... | bash, you should do bash -c "$(curl ...)"

The former is truly vulnerable, for the exact reason you mention, while the latter isn't.

> With curl | bash, the URL of the bash script is right there. I can copy it, plug it into my browser, and inspect with ease the shell script that will run

Not exactly, see http://thejh.net/misc/website-terminal-copy-paste

More savvy users might have set their shell up in a way that mitigates this risk, and some users will, as you mentioned, copy and paste URL, but go ahead and try that with the command in the link I provided - note that if you copy just the git URL, it only copies that bit, and then you can paste it into your browser, and not realize that there's a malicious bit before it

Of course, copying and pasting a command involving apt/dnf/etc is also vulnerable to that same attack :)

'bash -c $(curl )' doesn't mitigate a website's ability to detect whether you're downloading a file via curl versus accessing it in a browser (e.g. if (headers.USER_AGENT.includes("curl")) {}).

It does mitigate a different issue whereby the website can detect if you're piping the output of 'curl' directly to bash.

The safest option is definitely --> curl to file, audit what you downloaded locally, not in a browser, then bash -c that downloaded file.

But, again, this only really mitigates attacks from sketchy websites. If the website is trustworthy; both the website you're getting the curl | bash download string from, and the website the script is hosted at, such as Github, where 90% of these scripts are hosted; its not really mitigating a legitimate threat.

The copy-paste thing is interesting, but frankly: That's an argument for why the web is insecure, and that insecurity impacts every single method of software distribution which touches it. Copy-pasting a 'sudo apt-get install' line is the same class of risk here as copy-pasting a 'curl | bash' or 'bash -c $(curl)' line; you could swap the script to a new URL, or you could have them install a malicious package from apt, or even just nuke the apt-get install part and just do a curl | bash anyway. Definitely something that's great to know and educate about, but its not the strongest argument against curl | bash.