Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dear GitHub: no YAML anchors, please

blog.yossarian.net

Post reply on HN
Print view View original

71–80 of 159 posts

Re: Dear GitHub: no YAML anchors, please

#71

Post by woodruffw »

VGHN7XDuOXPAzol wrote: post #64
woodruffw wrote: post #53

Earlier quoted context omitted.

The flattening out is the problem: most (all?) widely used YAML parsers represent the YAML document using the JSON object model, which means that there's no model or element representation of the anchors themselves. That in turn means that there's no way to construct a source span back to the anchor itself, because the parsed representation doesn't know where the anchor came from (only that it was flattened).

This is something that a custom parser library could figure out, no? The same as how you have format-preserving TOML libraries, for instance. I think it makes way more sense for GitHub to support YAML anchors given they are after all part of the YAML spec. Otherwise, don't call it YAML! (This was a criticism of mine for many years, I'm very glad they finally saw the light and rectified this bug)

> This is something that a custom parser library could figure out, no? The same as how you have format-preserving TOML libraries, for instance.

Yes, it's just difficult. The point made in the post isn't that it's impossible, but that it significantly changes the amount of of "ground work" that static analysis tools have to do to produce useful results for GitHub Actions.

> I think it makes way more sense for GitHub to support YAML anchors given they are after all part of the YAML spec. Otherwise, don't call it YAML! (This was a criticism of mine for many years, I'm very glad they finally saw the light and rectified this bug)

It's worth noting that GitHub doesn't support other parts of the YAML spec: they intentionally use their own bespoke YAML parser, and they don't have the "Norway" problem because they intentionally don't apply the boolean value rules from YAML.

All in all, I think conformance with YAML is a red herring here: GitHub Actions is already its own thing, and that thing should be easy to analyze. Adding anchors makes it harder to analyze.

Re: Dear GitHub: no YAML anchors, please

#72

Post by VGHN7XDuOXPAzol »

jayd16 wrote: post #60
pacoWebConsult wrote: post #12

Can YAML go away entirely and instead allow pipelines to be defined with an actual language? What benefits does the runner-interpreted yaml-defined pipeline paradigm actually achieve? Especially with runners that can't be executed and tested locally, working with them is a nightmare.

Why do we think an arbitrary language is easier to reason about? If it was so easy you could just do it now. The yaml could be extremely simple and just call into your app, but most don't bother. I'm certainly willing to believe that yaml is not the ideal answer but unless we're comparing it to a concrete alternative, I feel like this is just a "grass is always greener" type take.

Is it actually possible to just have the YAML that calls into your app today, without losing the granularity or other important features?

I am not sure you can do this whilst having the granular job reporting (i.e. either you need one YAML block per job or you have all your jobs in one single 'status' item?) Is it actually doable?

Re: Dear GitHub: no YAML anchors, please

#73

Post by woodruffw »

VGHN7XDuOXPAzol wrote: post #69
woodruffw wrote: post #62

Earlier quoted context omitted.

No; GitHub shouldn't support YAML anchors because it's a deviation from the status quo, and the argument is specifically that the actions ecosystem doesn't need to make analysis any harder than it already is. (As the post notes, neither I nor GitHub appears to see full compliance with YAML 1.1 to be an important goal: they still don't support merge keys, and I'm sure they don't support all kinds of minutiae like non-…

> Conforming to a complex specification is not inherently a good thing Kind of a hard disagree here; if you don't want to conform to a specification, don't claim that you're accepting documents from that specification. Call it github-flavored YAML (GFY) or something and accept a different file extension. https://github.com/actions/runner/issues/1182 > YAML 1.1 to be an important goal: they still don't support merge k…

> Call it github-flavored YAML (GFY) or something and accept a different file extension.

Sure, I wouldn't be upset if they did this.

To be clear: there aren't many fully conforming YAML 1.1 and 1.2 parsers out there: virtually all YAML parsers accept some subset of one or the other (sometimes a subset of both), and virtually all of them emit the JSON object model instead of the internal YAML one.

Re: Dear GitHub: no YAML anchors, please

#74

Post by WorldMaker »

nirvdrum wrote: post #41
TheDong wrote: post #14

Earlier quoted context omitted.

Wanna DRY out your github actions yaml? Generate it from Dhall, or cue, or python, or some real language that supports actual abstractions. If your problem is you want to DRY out yaml, and you use more yaml features to do it, you now have more problems, not fewer.

Having used CI systems and application frameworks that support YAML anchors for configuration, adding in a programming language would be a massive amount of complexity for very little gain. We're not talking about dozens of locations with hundreds of lines of shared code. Asking the team to add a new build dependency, learn a new language, and add a new build step would create considerably more problems, not fewer. U…

Use the language you are already working in? Most languages have good YAML serialization and I think in most languages a function call taking a couple parameters that vary to produce slightly different but related objects is going to be as readable or more readable than YAML anchors.

Re: Dear GitHub: no YAML anchors, please

#75

Post by esafak »

TheDong wrote: post #14
anttiharju wrote: post #7

I think YAML anchors in GitHub Actions are very welcome, for example for DRYing the push/pull_request 'paths:' filters [1]. Now only if they supported paths filter for `workflow_call` [2] event in addition to push/pull_request and my life would be a lot easier. Nontrivial repos have an unfortunate habit of building some sort of broken version of change detection themselves. The limit of 20 unique workflow calls is qu…

Wanna DRY out your github actions yaml? Generate it from Dhall, or cue, or python, or some real language that supports actual abstractions. If your problem is you want to DRY out yaml, and you use more yaml features to do it, you now have more problems, not fewer.

If we're going there let's just stay there and not translate it back to YAML, which is absolutely inappropriate for specifying CI pipelines.

Re: Dear GitHub: no YAML anchors, please

#76

Post by woodruffw »

imcritic wrote: post #70

I think the author is nuts. First, he can just not use the feature, not advocate for its removal. Second, his example alternative is wrong: it would set variables for all steps, not just those 2, he didn't think of a scenario where there are 3 steps and you need to have common envs in just 2 of them.

The author does not think he's nuts :-)

> First, he can just not use the feature, not advocate for its removal.

I maintain a tool that ~thousands of projects use to analyze their workflows and actions. I can avoid using anchors, but I can't avoid downstreams using them. That's why the post focuses on static analysis challenges.

> Second, his example alternative is wrong: it would set variables for all steps, not just those 2, he didn't think of a scenario where there are 3 steps and you need to have common envs in just 2 of them.

This is explicitly addressed immediately below the example.

Re: Dear GitHub: no YAML anchors, please

#77

Post by esafak »

jayd16 wrote: post #60
pacoWebConsult wrote: post #12

Can YAML go away entirely and instead allow pipelines to be defined with an actual language? What benefits does the runner-interpreted yaml-defined pipeline paradigm actually achieve? Especially with runners that can't be executed and tested locally, working with them is a nightmare.

Why do we think an arbitrary language is easier to reason about? If it was so easy you could just do it now. The yaml could be extremely simple and just call into your app, but most don't bother. I'm certainly willing to believe that yaml is not the ideal answer but unless we're comparing it to a concrete alternative, I feel like this is just a "grass is always greener" type take.

> If it was so easy you could just do it now.

Some do just that: dagger.io. It is not all roses but debugging is certainly easier.

Re: Dear GitHub: no YAML anchors, please

#78

Post by VGHN7XDuOXPAzol »

woodruffw wrote: post #71
VGHN7XDuOXPAzol wrote: post #64

Earlier quoted context omitted.

This is something that a custom parser library could figure out, no? The same as how you have format-preserving TOML libraries, for instance. I think it makes way more sense for GitHub to support YAML anchors given they are after all part of the YAML spec. Otherwise, don't call it YAML! (This was a criticism of mine for many years, I'm very glad they finally saw the light and rectified this bug)

> This is something that a custom parser library could figure out, no? The same as how you have format-preserving TOML libraries, for instance. Yes, it's just difficult. The point made in the post isn't that it's impossible, but that it significantly changes the amount of of "ground work" that static analysis tools have to do to produce useful results for GitHub Actions. > I think it makes way more sense for GitHub t…

> conformance with YAML

maybe, but not entirely sure. 'Two wrongs don't make a right' kind of thinking on my side here.

But if they call it GFY and do what they want, then that would probably be better for everyone involved.

> they don't have the "Norway" problem because they intentionally don't apply the boolean value rules from YAML.

I think this is YAML 1.2. I have not done or seen a breakdown to see if GitHub is aiming for YAML 1.2 or not but they appear to think that way, given the discussion around merge keys

--

(though it's still not clear why flattening the YAML would not be sufficient for a static analysis tool. If the error report references a key that was actually merged out, I think users would still understand the report; it's not clear to me that's a bad thing actually)

Re: Dear GitHub: no YAML anchors, please

#79

Post by saurik »

woodruffw wrote: post #33
GuinansEyebrows wrote: post #31

Earlier quoted context omitted.

is your criticism leveled at yaml anchors or github? in my anecdotal experience, yaml anchors were a huge help (and really, really not hard to grasp at a conceptual level) in maintaining uniform build processes across environments.

It is specifically leveled at YAML anchors in GitHub. I don't have a super strong opinion of YAML anchors in other contexts. (This post is written from my perspective as a static analysis tool author. It's my opinion from that perspective that the benefits of anchors are not worth their costs in the specific context of GitHub Actions, for the reasons mentioned in the post.)

"YAML" should mean something. When I saw GitHub Actions supported "YAML", I thought "OK, certainly not my favorite, but I can deal with that", and so I read the YAML specification, saw anchors, and then had to realize the hard way that they didn't work on GitHub Actions, leaving me unsure what even would or wouldn't work going forward. Is this even the only way it differs? I don't know, as they apparently don't use YAML :/.

This also means that, if you use an off-the-shelf implementation to parse these files, you're "doing it wrong", as you are introducing a parser differential: I can put code in one of these files that one tool uses and another tool ignores. (Hopefully, the file just gets entirely rejected if I use the feature, but I do not remember what the experience I had was when I tried using the feature myself; but, even that is a security issue.)

> Except: GitHub Actions doesn’t support merge keys! They appear to be using their own internal YAML parser that already had some degree of support for anchors and references, but not for merge keys.

Well, hopefully they also prioritize fixing that? Doing what GitHub did, is apparently still doing, and what you are wanting them to keep doing (just only in your specific way) is not actually using "YAML": it is making a new bespoke syntax that looks a bit like YAML and then insisting on calling it "YAML" even though it isn't actually YAML and you can neither read the YAML documentation nor use off-the-shelf YAML libraries.

Regardless, it sounds like your tool already supports YAML anchors, as your off-the-shelf implementation of YAML (correctly) supports YAML anchors. You are upset that this implementation doesn't provide you source map attribution: that was also a problem with C preprocessors for a long time, but that can and should be fixed inside of the parser, not by deciding the language feature shouldn't exist because of library limitations.

Re: Dear GitHub: no YAML anchors, please

#80

Post by theamk »

phanimahesh wrote: post #46
TheDong wrote: post #14

Earlier quoted context omitted.

Wanna DRY out your github actions yaml? Generate it from Dhall, or cue, or python, or some real language that supports actual abstractions. If your problem is you want to DRY out yaml, and you use more yaml features to do it, you now have more problems, not fewer.

This. So true. Yaml has always been an overly complicated format, with weird quirks ( like norway becoming false in a list of country codes ). I find it an absolute shame that languages like Dhall did not become more popular earlier. Now everything in devops is yaml, and I think many developers pick yaml configs not out of good reasons but defaulting to its ubiquity as sufficient.

Why introduce all-new langugage, like Dhall, just for configuration? This seems like a total waste of time. And you still need to use "real" language (or bash) to write glue to connect to github configuration.

The config generators are very simple, and should to be written in whatever language your developers already know - which likely means Python or Javascript or Go.

Post reply on HN