He did not doubt fatality numbers, he made the point that one man cannot kill millions alone. It is something that requires the resources of a state. One man didn't kill millions, the state apparatus of Nazi Germany did.
There were numerous pogroms of Jews in European history but they pale in comparison to the Holocaust because it was the policy of an advanced, centralised, bureaucratic state.
Whether you support a big state or not, one undeniable fact about them is that they are highly capable. That capability can be used to do evil. cf. The US federal government of the 1890s, for example, was institutionally incapable of genocide.