Live data from Hacker News

Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata

salsa.debian.org

31–33 of 33 posts

Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata

#31
post #12

Earlier quoted context omitted.

If you are confident that certain build steps are deterministic, can you enable Dockerfile-like caching for intermediate steps? The way docker does it is take a hash of the "input" filesystem and the command, and see if there's an associated "result" filesystem, and if there is then just jump to evaluating the next command with the previous result as input.

That's precisely what bazel does (and remote build execution systems like buildbarn allow this cache to be effectively shared among users)

Ah, cool! IIRC Bazel predates docker, so perhaps it's more correct to say that dockerfiles use a Bazel-like command caching strategy.

Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata

#32
post #12

Earlier quoted context omitted.

That's precisely what bazel does (and remote build execution systems like buildbarn allow this cache to be effectively shared among users)

Ah, cool! IIRC Bazel predates docker, so perhaps it's more correct to say that dockerfiles use a Bazel-like command caching strategy.

Yep, moreover bazel has a fine grained dependency graph and thus can invalidate only a minimal part of the action graph when some inputs change.

Docker instead has to re-build all subsequent layers when the input of even just one layer changes.

The two tools sit on a different point in the spectrum of simplicity of use though. Maintain build files for bazel (and dealing with other constraints of hermetic execution) is hard time-consuming and it's hard to convince many teams it's worth the effort.

Docker apparently struck a sweet spot in that it's easy to explain how to craft linear build steps and it does a half-decent job in actually caching stuff. Sometimes it doesn't work well depending on your workflow but people then have the incentive to read about how to improve the "cacheability" of their dockerfiles (multistage, reorder, dockerignore, ...). As many things in our craft, the human aspect trumps over technical brilliance.

Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata

#33
nixbuild.net ("Nix Builds as a Service") can run any Nix build with fs randomization similar to disorderfs:

https://blog.nixbuild.net/posts/2021-01-13-finding-non-deter...

It isn't enabled by default but can be turned on with a setting:

https://docs.nixbuild.net/settings/#inject-fs-randomness

Post reply on HN