Can you explain how making flakey tests, not flakey, helps find bugs. I would have thought these differences are essentially free fuzzing and desirable?
Deterministic Linux for controlled testing and software bug-finding
21–30 of 71 posts
Re: Deterministic Linux for controlled testing and software bug-finding
#22TL;DR: This is a Rust project that forces deterministic execution of arbitrary programs and acts like a reproducible container. That is, it hermetically isolates the program from sources of non-determinism such as time, thread interleavings, random number generation, etc. Guaranteed determinism is a powerful tool and it serves as a basis for a number of applications, including concurrency stress testing, record/repla…
> AMA! Eager to try it but encountering the build error here - https://github.com/facebookexperimental/hermit/issues/11 Do you have a reference build log / environment you can share? Last known good commit sha and/or output from "rustup show"?
We're working on it! Should be fixed soon.
Re: Deterministic Linux for controlled testing and software bug-finding
#23Re: Deterministic Linux for controlled testing and software bug-finding
#24Earlier quoted context omitted.
> AMA! Eager to try it but encountering the build error here - https://github.com/facebookexperimental/hermit/issues/11 Do you have a reference build log / environment you can share? Last known good commit sha and/or output from "rustup show"?
We're working on it! Should be fixed soon.
Reverted a badly-timed breaking change that came through the sync system. Will fix it properly shortly (and add a Dockerfile and release tag). But for now you may have better luck on the main branch after that reversion, which yielded 6cb5575ffd287289769144ec82e2900cbf6cd1ad.
Let's discuss further on that issue #11.
Re: Deterministic Linux for controlled testing and software bug-finding
#25Earlier quoted context omitted.
It’s great how much open source infrastructure software has come out of FAANG in the past decade. Between Kubernetes, react, Kafka, etc it’s wild how much of my tech stack is open source software developed by people at large tech companies. It’s also great PR for the companies.
Heh, I implore you to consider the engineers-eye view. We're tech geeks inside or outside of FAANG, with all the usual incentives. I've been the tech lead on this project for 5 years, through 2 companies, and of course I hope someone finds it useful and chooses to contribute. I'd like to think we could help someone somehow with public relations, but I don't think we can ;-). Actually, I don't think any of the big tec…
Hey, I totally rewrote my comment after reading your response. Thanks for engaging - hearing from the project author made me realize I was acting like a cynical asshole. Congrats on Hermit and thanks for supporting free software.
Re: Deterministic Linux for controlled testing and software bug-finding
#26Can you explain how making flakey tests, not flakey, helps find bugs. I would have thought these differences are essentially free fuzzing and desirable?
Once we have complete control over the determinism of a test, we can start to play with tweaking the non-deterministic inputs in a controlled way. For example, we can tweak the RNG seed used for thread scheduling to explore schedules that wouldn't normally happen under the Linux scheduler.
Re: Deterministic Linux for controlled testing and software bug-finding
#27This has been the culmination of several years of work intercepting and sanitizing the Linux system call API. It's now open source.
Hey Ryan, just wanted to say I hope you're doing great these days! Really glad to see that the work originally done by you and Joe at Cloudseal evolving and becoming more readily available to all of us. I still thought back every once in a while about what y'all were up to. :) Super excited that it's kept on chuggin' and now is something we can all enjoy!
Re: Deterministic Linux for controlled testing and software bug-finding
#28Neat! This is the direction I’d hoped to see gvisor go in. What’s the reasoning for building from scratch and not piggybacking off gvisor?
We certainly looked into gVisor and Firecracker when we started this project a few years ago. These systems use KVM and gVisor in particular uses the Model Specific Registers (MSRs) to intercept system calls before forwarding them to the host kernel. Intercepting syscalls this way has less overhead than ptrace and we would have complete control over the system environment. I think it's a good approach and worth exploring more, but ultimately the deal breaker was that KVM requires root privileges to run and it wouldn't run on our already-virtualized dev machines. We also wanted to allow the guest program to interact with the host's file system. So, we went with good ol' ptrace. Last I checked gVisor also has a ptrace backend, but it wasn't very far along at the time. When going the ptrace route, there is less reason to depend on another project. Another reason of course is that we'd be beholden to a Google project. ;)
Re: Deterministic Linux for controlled testing and software bug-finding
#29Great work and thanks for making it OSS! I was familiar with the prior (academic) work and its limitations, specifically TCP/IP. Could you elaborate on how you solved that problem?
Re: Deterministic Linux for controlled testing and software bug-finding
#30Earlier quoted context omitted.
This is exactly the type of thing I've been wanting for testing my chess engine. Parallelism is based on emergent pseudorandom effects of things like interleaving causing searcher threads to mostly end up in non-overlapping parts of the search tree. One question: How do you avoid the program being affected by things like overall system load and memory pressure?
Since CPU is a "compressible" resource, system load doesn't affect the determinism. It'll make it slower of course. Since memory is a non-compressible resource, things can start getting killed by the OOM-killer and there's nothing we can do about it. There are also certainly things like external network communication and file system access that are non-deterministic that must be handled at a higher level (e.g., with…
The idea of CPU being compressible is very insightful, thanks.
I'm curious about what happens with time control though. Engines can be given a time constraint and will use various heuristics to allocate that time. How does Hermit intercept gettimeofday()?