I think the variety in configurations is completely irrelevant.
If you aren't using at least one accurate configuration for your testing, there is a huge risk that you miss your performance target by a huge margin. Your 60fps game could end up running at 20fps on your target minimum hardware. Small performance inaccuracies can massively add up if you have a non-emulated cache-miss or branch miss-predict delay in your inner-most loop.
I think you are massively overestimating how accuracy of timings that you can get though virtualisation or semi-accurate emulation. Yes. They are probably accurate enough for running any historic software from the era, as most code for the PC is well-behaved to not do the wrong thing when running too fast.
It's just for the use-case of developing new software, as soon as you start optimising or profiling, you need accurate timings. And yes, we might be talking about weird demo-scene style projects along the lines of "I want to get the absolute best possible graphics out of the computer I had 25 years ago, no frames dropped, no wasted cpu cycles". I'm talking about the kind of project were someone is writing inner loops with intrinsics or in assembly.
You might argue that such a project is a massive edge case that it's not worth catering too. And if you are writing an emulator, that's a 100% legitimate position to hold, emulators shouldn't have to cater for every possible usecase. My point is only "If you don't have a 100% accurate emulator, and there is some niche use case it can't emulate, then the hardware isn't fully preserved" and that it would be nice if an accurate emulator existed.
> Now on consoles I can see the benefit. Consoles are lots of identical hardware, operating systems that tend to get out of the way, and the people who develop for them only test (for obvious reasons) on the console hardware itself or at most a developer edition which has the same hardware (for obvious reasons again).
Yes, I've chased after bugs in console emulation (Dolphin Emulator) that were impossible to fix correctly without significantly more accurate emulation.
Like the game which memset a staging buffer before data had finished DMAing out. The game was only saved on real hardware because after memsetting, it invalidated the cachelines and in typical situations, none of the memset cachelines had been evicted. Impossible to correctly fix without emulating the existence of an L2 cache. We eventually resorted to patching the game to fix the bug.
Or games where video decoding stutters, because it has a hot inner loop that push the out-of-order CPU and has very few cache misses. It executes faster over the whole frame on real hardware than Dolphin's CPU timing model, which assumes a certain number of cache misses. The game must have tuned it's video codec to use as much CPU time as possible.
We have speed running tricks that don't work in Dolphin, because the depend on lagging the game. And games that freak out when the GPU executes too fast, but when you adjust the timings for those other games freak out because the GPU is executing too slow. It's impossible to calculate accurate GPU timings without running much of vertex transform and a basic depth rasterizer.
These are projects I'd love to work on at some point, accurate CPU and GPU timings for Dolphin, even if they don't run at full speed and bus contention is still ignored. I think might be possible to get within the correct order of magnitude (so 10-50% of realtime), which is workable for some usecases like TASes and testing bugs.