Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Helm local code execution via a malicious chart

github.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: Helm local code execution via a malicious chart

#91

Post by JohnMakin »

deathanatos wrote: post #84
JohnMakin wrote: post #73

Earlier quoted context omitted.

This isn't a uniquely helm thing though, they mostly use modified go templating. Lots of other things do this with yaml as well.

… and I think I'd argue that the parent's argument against the tooling would apply equally as well to those "other things", too. The alternative here is something that manipulates the data structure directly . E.g., it might permit me to say: my_config_map.data["key"] = some_string_value (This is in some pseudo-imperative language, vs. the parent's Lisp, but that distinction isn't particular relevant to the core of t…

I wrote a small terraform wrapper around helm provider that basically does what you’re saying. official kubernetes + tf support is poor, but it’s been working well for me. I rarely if ever have to touch the yaml templates that I maintain.

however, this is usually true with working with helm in general if you are using charts other people maintain. That’s one of the strengths of helm. you just shove your values into the chart and it should work. Maintaining charts is not fun though which is why I wrote the wrapper for my purposes.

Re: Helm local code execution via a malicious chart

#92

Post by kubectl_h »

akvadrako wrote: post #87
kubectl_h wrote: post #80

Earlier quoted context omitted.

I'm a dev that jumped to devops and one of my pet peeves will always be the lengths devops engineers go to avoid using a real programming language. Instead of interacting with all these APIs through python, ruby, lua, go, whatever they would rather build hodgepodge systems in bash, coreutils, curl (or wget. or both!) and jq (which is the worst). Or in the case of helm, just creating a half yaml/half Go SDK for genera…

You'll often find that if you write ops scripts in, say Python, it's largely calling external commands. When that's the case, bash if often the better choice, especially if you know it well. It has an excellent REPL, is easy to trace and is already installed everywhere.

> You'll often find that if you write ops scripts in, say Python, it's largely calling external commands.

That's a totally fine trade off for actual sane array/list functionality, robust string manipulation etc. I'd rather form shell commands in a programming language than in bash. People seem to love it.

I do not think for a second though that the average person that "knows bash well" can read and comprehend a multi hundred line bash script written by someone else as fast or even correctly has a seasoned python dev reading python written by someone else.

Re: Helm local code execution via a malicious chart

#93

Post by deathanatos »

ajross wrote: post #89
deathanatos wrote: post #83

Earlier quoted context omitted.

You're not installing the untrusted tarball; helm is merely supposed to be extracting it, and then rendering the templates contained within. (Those templates, once rendered, might then refer to pods, etc. that might be put into a k8s cluster (or perhaps we merely render then YAML, and never `apply` it), and in that sense, one might imagine that that is an install, but that's not the security boundary being crossed he…

You're quibbling over the meaning of "install" but apparently conceding the part about untrusted ? OK, fair enough. I still argue that any process involving the extraction and (ahem) "rendering of contained templates" from untrusted sources is broken in ways a fix for this particular symlink issue isn't going to address.

Yes? I don't find that that odd.

Certainly, it would be better to trust the upstream completely, but let's not kid ourselves? See the entire current state of software supply chain in the industry.

But when I visit a website, I don't expect the website to LCE me. Why should turning a YAML adlib into YAML LCE me, regardless of the trust of the upstream. This is not a privilege I'm expecting to give the upstream ever, and this behavior is a clear security bug, to me…

Re: Helm local code execution via a malicious chart

#94

Post by kubectl_h »

szszrk wrote: post #90
kubectl_h wrote: post #80

Earlier quoted context omitted.

I'm a dev that jumped to devops and one of my pet peeves will always be the lengths devops engineers go to avoid using a real programming language. Instead of interacting with all these APIs through python, ruby, lua, go, whatever they would rather build hodgepodge systems in bash, coreutils, curl (or wget. or both!) and jq (which is the worst). Or in the case of helm, just creating a half yaml/half Go SDK for genera…

> the lengths devops engineers go to avoid using a real programming language OK, but, you know... those tools were created by literal devs. Not in yaml, in a "real language"! So apparently devs thought they needed that. The argument should be towards all people - we love creating new abstractions and "simplify things", but we suck at honest evaluation of the impact of our creations. Still: I absolutely hate Helm temp…

> OK, but, you know... those tools were created by literal devs.

All `kubectl` does is create REST API calls to the control plane. So to be fair, what I'm grousing about can be accomplished just fine by a developer like me constructing API calls from python to update objects in k8s.

The problem is I work in devops where tooling written in proper languages with standard libraries that have things like useful arrays or robust string manipulation or ergonomic concurrency is a non-starter for some reason. The argument against that being mostly "I have to install the interpereter first".

Re: Helm local code execution via a malicious chart

#95

Post by szszrk »

kubectl_h wrote: post #94
szszrk wrote: post #90

Earlier quoted context omitted.

> the lengths devops engineers go to avoid using a real programming language OK, but, you know... those tools were created by literal devs. Not in yaml, in a "real language"! So apparently devs thought they needed that. The argument should be towards all people - we love creating new abstractions and "simplify things", but we suck at honest evaluation of the impact of our creations. Still: I absolutely hate Helm temp…

> OK, but, you know... those tools were created by literal devs. All `kubectl` does is create REST API calls to the control plane. So to be fair, what I'm grousing about can be accomplished just fine by a developer like me constructing API calls from python to update objects in k8s. The problem is I work in devops where tooling written in proper languages with standard libraries that have things like useful arrays or…

Now we get to the point - you are unhappy with how your org/team handles it. Would that be better description? Don't put all DevOps in one basket. It's a messy term anyway.

Many orgs and many teams allow that. That's why alternatives to Helm exist like cdk8s, or Pulumi for rest of infra.

For Kubernetes there is such a variety of tools, it's overwhelming. The only problem to tackle is that Helm/Argo is so popular, that alternatives are hard to sell regardless of real benefits.

Look at how big is a scene of Operators for k8s - this is where the programmable part of k8s went. You configure operators with their own CRDs (usually) that can be static/plain declarative text. Then you write actual code of the operator that deals with all the complexity of enforcing that state. For me this is where typically what you are talking about goes, while high level "non programmable" code sits in yaml in a other repo for others to maintain. Maybe this is where you should also go, standardize your work for others to consume in a form of operator?

I fully get neglect in adopting more complex tooling. sh, curl, sed, awk... those things are present almost everywhere and it's not that hard to get lucky and make a script that will run on almost everything your org has. And it actually might be fine for a decade or so.

I myself could not do code (even scripting) because one of my companies literally treated scripting in anything as development which was strictly regulated (so forbidden for any non dedicated dev role). Or an org that had not a single server in whole DC that has Python 3, years after it's release. Or more recent: some damn Ubuntu LTS that can't be easily upgraded to just 2-3 minor versions that this cool k8s library uses. Maintaining python versions on VMs is a pain in the ass, especially if your org has strict controls. Internet access to pip is not granted as well. UV gets the job done nowadays, if it's allowed, but long story short: that "fear of real language" can be as much lack of knowledge/skills as pragmatism that came from painful experiences.

Re: Helm local code execution via a malicious chart

#96

Post by moondev »

akvadrako wrote: post #88
ivan4th wrote: post #58

Helm is an abomination, as the whole idea of using a text template engine to generate YAML is. And this vulnerability adds insult to injury ;) Sorry, just can't really recover from trauma of counting spaces and messing up newlines, etc. when writing Helm templates. You know, Lisp "sucks" because "you need to count parenthesis" (you actually don't), yet Helm is a widely accepted technology where you need to count spac…

Something like kustomize was a better approach, where resources are templates semantically. Though it's lacking in several ways, like good destroy functionality.

I like wrapping with kapp for this. However kustomize still skips hooks if you inflate a chart that uses them and there is value in discoverability of helm apps installed with clear versioning

      kustomize build | kapp deploy -a my-app -y -f -

Re: Helm local code execution via a malicious chart

#97

Post by moondev »

JohnMakin wrote: post #73
ivan4th wrote: post #58

Helm is an abomination, as the whole idea of using a text template engine to generate YAML is. And this vulnerability adds insult to injury ;) Sorry, just can't really recover from trauma of counting spaces and messing up newlines, etc. when writing Helm templates. You know, Lisp "sucks" because "you need to count parenthesis" (you actually don't), yet Helm is a widely accepted technology where you need to count spac…

This isn't a uniquely helm thing though, they mostly use modified go templating. Lots of other things do this with yaml as well.

The funny thing is helm is as good or bad as what you make it. When folks complain about helm they are actually complaining about their own self created charts or poor selection of charts they install.

Re: Helm local code execution via a malicious chart

#98

Post by moondev »

galangalalgol wrote: post #56
immibis wrote: post #30

Earlier quoted context omitted.

NIH syndrome and "inverse second system effect". In the real second system effect, the second system is more complicated because it includes everything that could possibly be perceived as missing in the first system. In the inverse second system effect the first system was perceived as too complicated, not too simple, so the second system is much simpler and doesn't do its job well. Also this vuln has nothing to do w…

It is tangentially related in that yaml became normal to use as a DSL within the devops world. As another post said, everything becomes a DSL eventually because people want to be "fully configurable" not realizing that is roughly the same thing as not being complete yet. But in this case the lack of direct acknowledgement of yaml as an interpreted language with an interpreter that doesn't think of itself as such and…

This is like blaming python problems on yaml because someone embedded a python script in a multiline string.

Re: Helm local code execution via a malicious chart

#99

Post by galangalalgol »

moondev wrote: post #98
galangalalgol wrote: post #56

Earlier quoted context omitted.

It is tangentially related in that yaml became normal to use as a DSL within the devops world. As another post said, everything becomes a DSL eventually because people want to be "fully configurable" not realizing that is roughly the same thing as not being complete yet. But in this case the lack of direct acknowledgement of yaml as an interpreted language with an interpreter that doesn't think of itself as such and…

This is like blaming python problems on yaml because someone embedded a python script in a multiline string.

I wasn't blaming yaml at all. Our mistake is thinking we are using it as a configuration file. When we are actually using it as an interpreted language. Not yaml's fault people are writing dsl interpreters unknowingly. It's just related because people who make that mistake are picking yaml. I nearly made the mistake with toml a few years ago. You could even make the mistake with complicated environment variable usage. Whenever your configuration source is flexible enough to create executable primitives it needs to be sanitized. And really that is whenever a configurable value gets used in a conditional, which is often. Especially considering that even numeric values become conditional when they are used in operations that can result in ub or even just exceptions/panics/unhandled errors. Not a yaml exclusive.
Post reply on HN