It's hard to be scientific about this because there's no control group. If we clone your team and forbid them from using pair programming, in 5 years, which team will allow new developers to hit the ground faster? Which will have greater velocity. Which will have better code quality?
Some downsides to pair programming I've noticed:
1) People can tend to jump to the first solution they find, in order to not look slow or stupid. This doesn't allow for time to refactor the solution.
2) It can promote a culture of "spoken-word documentation" where all information about the system is conveyed via word-of-mouth, and nothing is recorded. You can see how this isn't scalable...
3) Not everyone has perfect internet, desk setup, comfortable chair, microphone quality, monitor resolution, etc. and it can lead to one person mumbling and another just going "yup uh-huh yeah"