> did we exclude drivers who were not eligible to drive
One could argue here that it would be unfair to exclude the worst performing humans. Plenty of humans cause accidents for easily preventable reasons, but that's an honest part of our behavior. Or to rephrase, I think excluding the categories of people who drive terribly, but do still drive nonetheless, would be less representative of reality.
> Did Waymo drive across all the same areas and time periods as the humans did with an equal "proportion"?
No, they didn't. Their methodology didn't adjust for things like highway vs city street miles, although it did adjust for city and state.
> Could we compare "time behind wheel" instead of miles travelled?
We could, but that would incentivize slow driving rather than safe driving. You need to travel the same minimum distance from a hypothetical "Point A" to "Point B" no matter what, but the number of minutes could be inflated easily. Economic factors discourage inflating the number of miles.
> There are so many ways to slice and dice this and come up with favourable results for self-driving systems that I can't but be dubious of any claim.
I think it's fair to criticize the fact that they didn't adjust for geographic area with as much granularity as they should have (i.e. for highways vs city streets, or for excluded zones like airports). But other than that, their methodology seems solid to me.