Testing Distributed Systems with Deterministic Simulation (2014) [video]
1–9 of 9 posts
Re: Testing Distributed Systems with Deterministic Simulation (2014) [video]
#2Discussed a bit at the time: https://news.ycombinator.com/item?id=8345030
Re: Testing Distributed Systems with Deterministic Simulation (2014) [video]
#3after the jepsen/mongodb post that hit front page nearly at the same time, it seems like the foundationdb team has the exact opposite values regarding resilience and data safety guarantees. I cant wait for the first results of the couchDb rewrite on foundationDb
Re: Testing Distributed Systems with Deterministic Simulation (2014) [video]
#4My experience is that this is difficult without the system under test being designed to operate deterministically.
So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both.
Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically.
A while ago I built a simulator testing harness for an early version of the Knative autoscaler[0]. I eventually ran aground on the difficulty of keeping up with changes, but it may be rearchitected and repurposed to assist Kubernetes HPA development instead.
FoundationDB's approach is to pretty much bake optional determinism into the language with some macro magic. It makes me wish that were easier to do in other languages.
Re: Testing Distributed Systems with Deterministic Simulation (2014) [video]
#5My experience is that this is difficult without the system under test being designed to operate deterministically. So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both. Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically. A while ago I built a simulator testing harn…
In the FDB simulator case, every event has a future time at which it will fire, and the clock advances to that time when the event is dequeued.
This also has the side benefit of artificially “speeding up time”, which is useful when many actions happen based on long timeouts.
Re: Testing Distributed Systems with Deterministic Simulation (2014) [video]
#6This is similar to https://github.com/microsoft/coyote/
Re: Testing Distributed Systems with Deterministic Simulation (2014) [video]
#7My experience is that this is difficult without the system under test being designed to operate deterministically. So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both. Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically. A while ago I built a simulator testing harn…
In the FDB simulator case, every event has a future time at which it will fire, and the clock advances to that time when the event is dequeued. This also has the side benefit of artificially “speeding up time”, which is useful when many actions happen based on long timeouts.
I used the same approach, which I learned from DES textbooks. I wrote up the design here: https://github.com/pivotal/skenario/blob/master/docs/concept...
Re: Testing Distributed Systems with Deterministic Simulation (2014) [video]
#8My experience is that this is difficult without the system under test being designed to operate deterministically. So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both. Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically. A while ago I built a simulator testing harn…
WebAssembly allows deterministic execution (with caveats: https://github.com/WebAssembly/design/blob/master/Nondetermi...) of any code. I think it is a perfect platform for such type of testing.
Re: Testing Distributed Systems with Deterministic Simulation (2014) [video]
#9My experience is that this is difficult without the system under test being designed to operate deterministically. So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both. Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically. A while ago I built a simulator testing harn…
I used to have a similar opinion, that it requires a special language to leverage such power. But recently I've discovered that Rust, with its latest advancement in async/await design together with swappable runtime, can provide a decent story for building determinism tests like described in the video. Yes of course one still needs to mock all the IO as well as clock part, but the deterministic core is already there, and with a carefully designed runtime and surrounding library, we might have pretty good support in Rust for this.
Disclaimer: I'm never a Rust zealot, I do understand Rust has its tradeoffs, and it's not the panacea for every problem. In fact I do have side projects which are perfectly suited and written in Go. I'm just saying Rust's design turns out to be suitable for such a deterministic testing structure, and Rust's target for system programming, can also benefit A LOT from this style of testing.
And also a shameless plug: I do have some initial work exploring this area: https://github.com/xxuejie/diviner. It's still quite rough and a lot of work is needed but I do believe this is something that is worth exploring.