Earlier quoted context omitted.
>If your test was to compare an intraframe between vp8 / baseline h264 / and Theora, you would have concluded Theora was the best by a wide margin. But it wasn't. I was comparing the visual quality of the whole video, and provided the full encoded clips for people to download and compare for that reason. I am willing to do further test encodes, but have no interest in doing something like encoding all 28 HD test clip…
I have no interest in doing something like encoding all 28 HD test clips, since as a purely visual comparison, it would be incredibly exhausting. Science is exhausting. If you're not working hard, then you're likely to miss the interesting (counter-intuitive) results. In fact, finding counter-intuitive results is the whole point of science. If the truth were intuitive, explanations wouldn't need testing.
Anyway, I brought up the subject to some Xiph folks over at IRC. Maybe in the future the test clips will come equipped with more detailed information to help in testing. It'd also benefit smaller scale tests, since it'd allow one to identify possible biases more easily.