Live data from Hacker News

What's the big deal about Deterministic Simulation Testing?

notes.eatonphil.com

11–15 of 15 posts

Re: What's the big deal about Deterministic Simulation Testing?

#11

I don't think you have to have systems in the same thread/process if you have bake in an API for controlling time and ingress/egress for each component. (depending on what you're trying to test) You can have the communication channels between components under the control of the simulation environment rather than have them happen in their 'normal' manner. This allows you to inject latency between components, 'fiddle'…

The reason for using a single underlying thread/process is to prevent the OS scheduler from interfering with deterministic execution. You can't control how and when the OS scheduler kicks in, nor can you perfectly reproduce the clock drift/jitter between multiple cores. If the program under test spawns threads, then you'll have to emulate the execution of those threads by writing your own scheduler whose time slicing…

You can control scheduling; that is how many record-replay based time-traveling debuggers do it.

Also, scheduling is independent of deterministic execution unless you are doing inherently non-deterministic things like multithreaded shared memory accesses which you can not simulate faithfully anyways. The only thing that matters in a deterministic execution model is runs of deterministic execution interrupted with non-deterministic events injected at precise points in the execution trace.

When serializing onto a single thread you already need to define some sort of correspondence between "simulated scheduler state" to number of instructions to execute as you are already giving up on the actual scheduler (unless you do not care about correspondence to the actual schedule configuration). You just do that, but you get to execute with all of your cores until you reach the injection point (which is how replay systems can work already). Now you can execute in parallel (multiprocessing only though, no multithreading) and use blocking I/O.

Re: What's the big deal about Deterministic Simulation Testing?

#12

Considering you actually have to design around DST, I’m still widely unconvinced that the time and effort spent setting up DST and fuzzing hoping it finds your bugs wouldn’t be better spent actually proving that your design is bug free using tools like TLA+ before intelligently using static analysis and formal proof during implementation. I believe DST to be the wrong solution to the actual problem. Its main advantag…

One of the big problems with using TLA+ is that it verifies your design, not your code. People are looking for ways to link the two. Formal proof works but is too expensive for most businesses.

The most promising approach I've seen so far is... DST! First we simulate a system, we generate a bunch of timelines, then we see if those timelines are valid behaviors in the TLA+ design. I've heard of a few success stories and it's definitely cheaper than formal proof!

Re: What's the big deal about Deterministic Simulation Testing?

#13
post #12

Considering you actually have to design around DST, I’m still widely unconvinced that the time and effort spent setting up DST and fuzzing hoping it finds your bugs wouldn’t be better spent actually proving that your design is bug free using tools like TLA+ before intelligently using static analysis and formal proof during implementation. I believe DST to be the wrong solution to the actual problem. Its main advantag…

One of the big problems with using TLA+ is that it verifies your design, not your code. People are looking for ways to link the two. Formal proof works but is too expensive for most businesses. The most promising approach I've seen so far is... DST! First we simulate a system, we generate a bunch of timelines, then we see if those timelines are valid behaviors in the TLA+ design. I've heard of a few success stories a…

> The most promising approach I've seen so far is... DST! First we simulate a system, we generate a bunch of timelines, then we see if those timelines are valid behaviors in the TLA+ design.

That’s just testing again. That’s not linking the code to the design.

DST feels to me like the way people treated memory access before Rust came around. Doing things properly was also seen as too costly then but now that it’s trendy everyone is fully behind lifetime tracking. Same here, formal proof is “too costly” but spray and pray approach like DST is somehow acceptable. Anyway, keeping with the Rust example I guess I just have to wait two decades and the next generation might finally see the light.

Re: What's the big deal about Deterministic Simulation Testing?

#14

I don't think you have to have systems in the same thread/process if you have bake in an API for controlling time and ingress/egress for each component. (depending on what you're trying to test) You can have the communication channels between components under the control of the simulation environment rather than have them happen in their 'normal' manner. This allows you to inject latency between components, 'fiddle'…

The reason for using a single underlying thread/process is to prevent the OS scheduler from interfering with deterministic execution. You can't control how and when the OS scheduler kicks in, nor can you perfectly reproduce the clock drift/jitter between multiple cores. If the program under test spawns threads, then you'll have to emulate the execution of those threads by writing your own scheduler whose time slicing…

This depends on what you're testing. If you're collapsing threads/processes into a single thread you're making decisions about the order of execution anyway so you're not going to catch errors introduced around unexpected preemption or inter-core process/thread timing issues.

If you're not looking for that then you can build your software/system to have known sync-points where you can allow a given process to stop/wait and allow other processing to occur. This can then be in-process/out of process/remote.

As you say, this ends up being a scheduler with which you have to employ knowledge about the execution/communication channels in order to coordinate correctly.

Post reply on HN