Earlier quoted context omitted.
If you are confident that certain build steps are deterministic, can you enable Dockerfile-like caching for intermediate steps? The way docker does it is take a hash of the "input" filesystem and the command, and see if there's an associated "result" filesystem, and if there is then just jump to evaluating the next command with the previous result as input.
That's precisely what bazel does (and remote build execution systems like buildbarn allow this cache to be effectively shared among users)
Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
31–33 of 33 posts
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#32Earlier quoted context omitted.
That's precisely what bazel does (and remote build execution systems like buildbarn allow this cache to be effectively shared among users)
Ah, cool! IIRC Bazel predates docker, so perhaps it's more correct to say that dockerfiles use a Bazel-like command caching strategy.
Docker instead has to re-build all subsequent layers when the input of even just one layer changes.
The two tools sit on a different point in the spectrum of simplicity of use though. Maintain build files for bazel (and dealing with other constraints of hermetic execution) is hard time-consuming and it's hard to convince many teams it's worth the effort.
Docker apparently struck a sweet spot in that it's easy to explain how to craft linear build steps and it does a half-decent job in actually caching stuff. Sometimes it doesn't work well depending on your workflow but people then have the incentive to read about how to improve the "cacheability" of their dockerfiles (multistage, reorder, dockerignore, ...). As many things in our craft, the human aspect trumps over technical brilliance.
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#33https://blog.nixbuild.net/posts/2021-01-13-finding-non-deter...
It isn't enabled by default but can be turned on with a setting: