[deleted]
So you think you want to write a deterministic hypervisor?
41–50 of 56 posts
Re: So you think you want to write a deterministic hypervisor?
#42Fun read. It has some similar ideas to https://dedis.cs.yale.edu/2010/det/ but that was actually focused on multicore processing and the communication across cores.
Congrats on the launch.
Re: So you think you want to write a deterministic hypervisor?
#43What a tease! They describe in detail two problems they had to “invent workarounds” for, but say nothing about what the workarounds are. I’m very curious, since both of the problems sound quite hard to work around. I wonder if they’re being purposefully vague to make it harder for competitors to replicate their work…
> I wonder if they’re being purposefully vague to make it harder for competitors to replicate their work…
"I’m necessarily leaving out a ton of detail, of course, both for the sake of brevity and competitive edge."
Re: So you think you want to write a deterministic hypervisor?
#44This is for a distributed database product that claims to bypass the CAP theorem or have I misunderstood? > Back then Spanner wasn’t public yet and a lot of people misinterpreted the CAP theorem to say that a strongly consistent database couldn’t also be highly available in the face of network faults. - https://antithesis.com/blog/is_something_bugging_you/
Probably referring to: https://cloud.google.com/blog/products/databases/inside-clou... https://apple.github.io/foundationdb/cap-theorem.html
Oh - it's just a standard CP database where, in the event of a partition, the partition knows it's the partition because it's smaller than a quorum and won't let you access it.
Re: So you think you want to write a deterministic hypervisor?
#45What a tease! They describe in detail two problems they had to “invent workarounds” for, but say nothing about what the workarounds are. I’m very curious, since both of the problems sound quite hard to work around. I wonder if they’re being purposefully vague to make it harder for competitors to replicate their work…
https://rr-project.org/ had the same problem. They use the retired conditional branch counter instead of instruction counter, and I believe then instruction stepping until at the correct address.
Re: So you think you want to write a deterministic hypervisor?
#46Earlier quoted context omitted.
x86 processors have a LOCK instruction prefix, which makes some instructions atomic including increment. Increment is nontrivial to make atomic, because there are two memory accesses: read X and write back X+1. It's a bit slow, because it has to inform every other core in the system, "Hey, I just read this memory address and I'm about to write something back to it, so don't don't use it until I'm done." C++ has funct…
The way that's currently implemented is actually just one memory operation these days. L2 (or really wherever coherency is mostly managed) has a tiny ALU so in addition to read or write, atomicop is an operation you can send to L2. It'll gain Modified or Exclusive access to the cache line(s) that op is addressed to and just do the operation right there. That way the normal cache protocol is all you need for atomicity…
That's not how it works on x86 as far as I know. The atomic ops are simply performed by the ALU, against the L1 cache when the instruction is about to retire. Atomicity is guaranteed by not allowing the line to be stolen by another core between the operation, and memory order is ensured by draining the store buffer before executing the op and other speculation (eg loads can speculatively pass even atomic operations).
Re: So you think you want to write a deterministic hypervisor?
#47Did you decide on a hypervisor instead of an emulator for overhead reasons? I don't work there but I heard Microsoft has something called tkofuzz which similarly emphasizes determinism but uses Bochs internally.
Re: So you think you want to write a deterministic hypervisor?
#48I love this. Many a moon ago, I worked on a system called Aikido at MIT, which combined a special built hypervisor with a binary rewriting system (DynamoRio) to enable efficient time travel debugging and race detection of parallel applications.
If anyone's interested, here's a publication that talks about it in more detail:
https://dspace.mit.edu/handle/1721.1/72082
The use of performance counters here also reminds me of another project I worked on called Kendo, which was a posix thread like replacement that used performance counters to enforce a deterministic interleaving of synchronization operations (mutexes, etc). The system could guarantee determinism for programs that didn't have race conditions. Back then, I found that counting instructions wasn't deterministic on the processors of the time, but counting store operations was. If anyone's interested in that work, here's the publication:
http://www.cag.csail.mit.edu/~mareko/asplos073-olszewski.pdf
Re: So you think you want to write a deterministic hypervisor?
#49I'm not familiar with the area, so am likely missing something, but how do they do deterministic thread-level context switching? Something like: var_1 = 0 var_2 = 0 thread_a: while true: something_complex() var_1 ++ thread_b: while true: something_complex() var_2 ++ Under the quoted definition of determinism, for every point in time, var_1 and var_2 should have the same values across all executions. But this would se…
You are exactly correct. Our hypervisor grants you this power (and then we use the power to explore as many possible values of var_1 and var_2 as we can, in case some of them trigger a concurrency bug, e.g.)
The resolution of your deterministic scheduler is probably not based on quanta like "instructions", right? More likely it takes traditional input like a stable clock tick?
Re: So you think you want to write a deterministic hypervisor?
#50I don't understand how this works in the case of testing many applications running on many machines, where many services on many machines need to communicate with each other. We deploy a mix of systemd services and OCI containers (running on podman and Docker) to different machines, the exact mix on each machine depends on the machine's intended purpose.
We currently run CI tests using QEMU VMs. These VMs comprise a few systems representative of those that we deploy to production.
Does adopting Antithesis mean that all non-containerized applications would need to be OCI-ified and every interaction would need to be mocked? There's a sort of combinatorial explosion that I'm concerned about when I'm thinking about testing/adding a new service to a system: All services on which it depends need to be mocked and all services which depend on it require creating a mocked version of it.
Seems like a lot of work. Can someone please help clarify things for me?
Also, how could we test the behavior of non-application code like drivers or the kernel itself?