Deterministic Linux for controlled testing and software bug-finding

Sorry, just an additional question as this project is so interesting:

Does Hermit support running different processes in parallel in certain situations?

It seems like this should be possible to do as long as they are appropriately isolated from each other.

Specifically, if two processes don't share any writable memory mappings then I'm thinking it might be possible for Hermit to exploit the underlying existing isolation between them so that they could run code in-between system calls in parallel (while still making sure that the actual system calls happen in a deterministic order).

Perhaps it would even be possible for the two processes to execute system calls in parallel if they are unrelated and they do not affect deterministic execution (such as if they are doing I/O to different files or directories, or one process doing I/O while another is doing something completely unrelated like getting the time, etc).

Although I guess this latter point (of executing syscalls in parallel) would require doing rewind/replay because it wouldn't be possible to know in advance whether the system calls are going to be related or not, as the two processes might not do the syscalls at exactly the same time.

Is Hermit doing this (executing code in-between syscalls in parallel, at least)? Or do you think it would be possible to do it?

This could significantly increase the usefulness of Hermit for distros that want to do deterministic builds of packages, as most compilation could happen in parallel / i.e. use several cores at the same time!

I'm thinking about huge package builds such as Chromium, Firefox, the Linux kernel, etc. which would perhaps take days to build if they were completely serialized into a single CPU core.

Wow, that's so cool!

That's actually much better than what I was thinking and if I understand you correctly, that should allow you to exploit practically all possible parallelism opportunities (given the constraints).

I'm guessing that your plan would require unmapping shared memory pages so that Hermit can catch memory accesses and request permission from the scheduler (to access the memory pages) when these memory accesses happen. But I think it should be possible to do it, yes (with some overhead if two threads/processes access shared memory pages simultaneously very often, I guess).

Awesome work, thank you!

Yeah, it's interesting to think about persisting the state we would need to make the file system more sympatico with Hermit. If we were willing to have a daemon.... Meta develops this "watchman" tool that our build infrastructure uses. I think for existing file systems we could imagine a daemon that watches the directory and caches what we need.

But if we could dream of new file systems, then I want one that is basically btrfs but with O(1) file-level parallel-hashes (rather than block level). Heck maybe it could even enforce a sorted order on directories for us too. The hashes would be very useful in record-and-replay scenarios, where you could know immediately whether the input files are in your content-addressible blob storage without having to hash them every time.

(We have some hash support in Meta's EdenFS FUSE file system https://github.com/facebook/sapling)

P.S. about Reproducible Builds -- we pitched this to the community in 2019 (at the Reproducible Builds Summit) and with that ASPLOS'20 paper, but we would be eager to see anyone wanting to pick it up and take it further.

There’s a bit of shared lineage there. Joseph Devietti and I started working in this area together around 2015. And Joe had worked on the deterministic dOS at UW during his PhD (which led to Jinx).

Another related effort is antithesis.com which also seems to use a hypervisor approach rather than Hermits Linux-syscall-API level approach.

Performance is much better than UndoDB I suppose? Are there any sources of nondeterminism UndoDB handles but hermit does not?