Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
1–10 of 33 posts
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#2On Gitlab isn't possible even to read the Readme without javascript enabled. Too bad!
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#3On Gitlab isn't possible even to read the Readme without javascript enabled. Too bad!
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#4Buildbarn, a build cluster implementation for Bazel that I maintain, can also run build actions (compilation steps, unit tests) in a FUSE file system. Though the primary motivator for this is that it reduces the time to construct a build action's file system to nearly instant, it has the advantage that I can also do things similar to disorderfs. Shuffling directory listings is actually something that I also added. Pretty useful!
https://github.com/buildbarn/bb-remote-execution/blob/eb1150...
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#5Sorry for plugging in. NixOS has a plan to use DisorderFS in making 100% reproducible build.
As I understand it, currently several packages are not reproducible (like, python, pytest, gcc), so it is not priority, but when those large packages will be done, r13y will start using DisorderFS to uncover remaining reproducibility bugs.
This is too idealistic, but gives lots of pleasure about package space.
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#6Sorry for plugging in. NixOS has a plan to use DisorderFS in making 100% reproducible build. https://r13y.com/ As I understand it, currently several packages are not reproducible (like, python, pytest, gcc), so it is not priority, but when those large packages will be done, r13y will start using DisorderFS to uncover remaining reproducibility bugs. This is too idealistic, but gives lots of pleasure about package spac…
Instead of introducing non-determinism, shouldn't it instead try to enforce determinism in any possible way? (E.g. running all processes under ptrace or by using virtualization and thereby making the OS behave in a deterministic way during a build).
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#7Buildbarn, a build cluster implementation for Bazel that I maintain, can also run build actions (compilation steps, unit tests) in a FUSE file system. Though the primary motivator for this is that it reduces the time to construct a build action's file system to nearly instant, it has the advantage that I can also do things similar to disorderfs. Shuffling directory listings is actually something that I also added. Pr…
If you are confident that certain build steps are deterministic, can you enable Dockerfile-like caching for intermediate steps? The way docker does it is take a hash of the "input" filesystem and the command, and see if there's an associated "result" filesystem, and if there is then just jump to evaluating the next command with the previous result as input.
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#8The work done in Debian around reproducible builds is really impressive.
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#9Sorry for plugging in. NixOS has a plan to use DisorderFS in making 100% reproducible build. https://r13y.com/ As I understand it, currently several packages are not reproducible (like, python, pytest, gcc), so it is not priority, but when those large packages will be done, r13y will start using DisorderFS to uncover remaining reproducibility bugs. This is too idealistic, but gives lots of pleasure about package spac…
Instead of introducing non-determinism, shouldn't it instead try to enforce determinism in any possible way? (E.g. running all processes under ptrace or by using virtualization and thereby making the OS behave in a deterministic way during a build).
I think the idea is you can build from source on any file system or drive and get the same results. By adding artificial purposeful nondeterminism you can fix your builds to account for unintentional natural nondeterminism.
Re: Disorderfs: FUSE-based filesystem that introduces non-determinism into metadata
#10Sorry for plugging in. NixOS has a plan to use DisorderFS in making 100% reproducible build. https://r13y.com/ As I understand it, currently several packages are not reproducible (like, python, pytest, gcc), so it is not priority, but when those large packages will be done, r13y will start using DisorderFS to uncover remaining reproducibility bugs. This is too idealistic, but gives lots of pleasure about package spac…
Instead of introducing non-determinism, shouldn't it instead try to enforce determinism in any possible way? (E.g. running all processes under ptrace or by using virtualization and thereby making the OS behave in a deterministic way during a build).
The goal is to ensure that builds can be deterministic despite non-determinism.
Once such criterion is enforced, then everybody can reproduce the build with that set of source files and build instructions, without requiring a special environment that forces a specific order of events.