Grep one-liners as CI tasks • Hacker News Board

Re: Grep one-liners as CI tasks

#31

Post by chubot » Sat, Jan 15, 2022, 8:07 AM UTC

If you think of a CI configuration as a shell script, then this is normal and not surprising.

A CI config is just a big job that invokes a bunch of tools like "go build" or "npm test", which is exactly what a shell script is too. I would get rid of the YAML and use shell for the whole thing :)

Shell does have some problems, like needing to be more parallel, incremental, and reproducible, but the YAML-based CI systems all have those problems too.

Related: http://www.oilshell.org/blog/2021/04/build-ci-comments.html#... (and the entire post)

Re: Grep one-liners as CI tasks

#32

Post by xorcist » Sat, Jan 15, 2022, 3:44 PM UTC

don-code wrote: post #20

xorcist wrote: post #19

Wouldn't quick sanity checks like these make more sense in a git push hook? No reason to wait for the CI. There's also the risk of broken intermediary commits unless the CI check each and every commit in isolation.

One of the benefits of CI is that I don't need to make assumptions about the environment I'm operating in - that is, I don't need to worry about a tool like `jq` being installed, whether I have GNU or BSD `date`, or the biggest one in this case, whether I've even installed a commit hook after cloning the repo. It only takes one new hire that hasn't set up the hooks to merge a change that makes the repo fail checks.

A post-receive hook runs on your git server, in an environment just as known to you.

It denies all pushes which doesn't meet the specified requirements with an error message, and the end user has to re-do the commit and push again.

This is likely how your authorization is done today already.

The alternative would be a pre-commit hook, but that runs when crafting the commit, and under the control of the end user. That can make for a better user experience since it runs even earlier in the process but isn't necessarily secure. Of course, one can have both.

Re: Grep one-liners as CI tasks

#33

Post by ilyash » Sat, Jan 15, 2022, 6:44 PM UTC

Interesting to see how nobody cares about exit code 2. Modify your grep command to have syntax error and you will always think you have all the translations.

Edit: ah.. forgot the whole point.. In Next Generation Shell (author here) this doesn't happen: ngs -e '$(grep ...).not()' does right thing. grep exit code 0 becomes 1, 1 becomes 0, 2 becomes 240 (exception)

Re: Grep one-liners as CI tasks

#34

Post by gravypod » Sun, Jan 16, 2022, 1:34 AM UTC

joatmon-snoo wrote: post #28

gravypod wrote: post #3

I think this is a great example of where build systems, that understand the need for testing, can help out. In a build system I use quite often (Bazel) you can express this as an `sh_test()` [0] which provides documentation about what you're attempting to do and provides you with a way to reproduce the failure locally. You don't have to push the code and wait for CI to fail to find out, or debug, this error. Extra fu…

Tests are a good way to assert an invariant that you expect of your codebase, but as with all things, resolving the error can get a bit tricky/frustrating. The canonical example in my mind is any kind of autofix-able linter, where there's some kind of patch (or more nuanced autofix) that the linter can generate on-the-spot for you. With a sh_test construct (or any other test), you generally find yourself printing out…

> (The worst offenders of this that I remember dealing with at Google were when generated code/artifacts had to be checked into the repo instead of getting being wired into the build system because and a large-scale change you were doing looked at one of them the wrong way...)

Been there. It's unfortunate because there are `genrule()`s for this very reason. But, definitely. Linting is very important and trunk looks great! But linting and config validation are very different use cases. At a previous company I had test that loaded a set of configs from repo and made sure all of them were valid with app level logic. It was crude but helpful. Trunk would be ideal.

Also, question, have you thought of writing your own linter+autoformatter with treesitter? Are you planning to expand into other tooling?

Re: Grep one-liners as CI tasks

#35

Post by joatmon-snoo » Tue, Jan 18, 2022, 11:54 PM UTC

vlovich123 wrote: post #29

joatmon-snoo wrote: post #28

Earlier quoted context omitted.

Tests are a good way to assert an invariant that you expect of your codebase, but as with all things, resolving the error can get a bit tricky/frustrating. The canonical example in my mind is any kind of autofix-able linter, where there's some kind of patch (or more nuanced autofix) that the linter can generate on-the-spot for you. With a sh_test construct (or any other test), you generally find yourself printing out…

The problem with automatic linting is that there’s no good answer. Either you just fail and ask the user to resubmit or you fix up the formatting on their behalf. If you do the latter, you’re likely also pushing to the review system meaning the code under review and your local version no longer match which causes problems when rebasing and whatnot. The secondary problem is that the lint tool chain running locally som…

Thanks for the feedback! All of those are very real problems and we've spent a lot of time thinking about how to handle them:

> Either you just fail and ask the user to resubmit or you fix up the formatting on their behalf.

We only apply autofixes when running on the user's machine; when running on CI, as you point out, we only show them because of Git's decentralized nature here.

> The secondary problem is that the lint tool chain running locally sometimes doesn’t version match what’s in CI which poses a set of problems too, although in practice I think that’s raised as a concern only by teams that don’t establish good hygiene of checking in the set of tools (maintenance of multiple platforms is real there though).

We've built out our own system for managing (mostly) hermetic toolchain installs, and require users to version their linters in their trunk.yaml configuration right now. (There are some open questions here around how we handle version skew between our toolchain version and the system-installed toolchain version.)

We're also well aware of the problems that emerge as you scale to a larger team; hierarchical configs are one of the things that we've discussed but are waiting until we have more traction to flesh out. (All of us working on the linter side have experience setting tech direction for orgs of 100+ engineers).

Re: Grep one-liners as CI tasks

#36

Post by joatmon-snoo » Wed, Jan 19, 2022, 12:02 AM UTC

gravypod wrote: post #34

joatmon-snoo wrote: post #28

Earlier quoted context omitted.

Tests are a good way to assert an invariant that you expect of your codebase, but as with all things, resolving the error can get a bit tricky/frustrating. The canonical example in my mind is any kind of autofix-able linter, where there's some kind of patch (or more nuanced autofix) that the linter can generate on-the-spot for you. With a sh_test construct (or any other test), you generally find yourself printing out…

> (The worst offenders of this that I remember dealing with at Google were when generated code/artifacts had to be checked into the repo instead of getting being wired into the build system because and a large-scale change you were doing looked at one of them the wrong way...) Been there. It's unfortunate because there are `genrule()`s for this very reason. But, definitely. Linting is very important and trunk looks g…

> But linting and config validation are very different use cases. At a previous company I had test that loaded a set of configs from repo and made sure all of them were valid with app level logic.

Understandably so, especially when you might need, say, an AWS permission to actually validate a config (or are there other distinctions between the two that I'm not thinking of?)

I wasn't previously aware of treesitter, but it definitely looks like the type of thing we eventually want to do!

One of the things we currently do is run our own daemon and LSP proxy, so that we can share functionality between the CLI version of our tool and the VSCode extension. Right now we treat most linters as CLI black boxes, but we're looking at moving to treat more of them as LSP servers (we currently do this with eslint, but don't yet with ones like clangd or tflint).

And there is indeed more that we're building out! The other product that we're actively prototyping (and dogfooding) is our merge queue: https://trunk.io/products/merge.

Long-term, the vision for Trunk is to make development overall a more seamless experience, by actually tying together the various disparate tools that solve individual problems that we run into.