In the beginning, smartphones were all tiny, because it was universally agreed that smaller cell phones were better.
Then there was a huge, neverending push to make phones as large as possible, or larger. Sometimes smaller phones have been offered as an unpopular option, but more often "enormous" is the minimum size.
And as long as that's been going on, people have been complaining that they want their phone to fit in their hand.
Stipulate that a majority seems to feel that there's no need for a phone to fit in your hand. Why are the manufacturers so insistent on not providing small phones? Shirts come in all different sizes. How much does it cost to design an additional size of phone?
(Related: ever since the switch to 16:9 laptop screens, everyone has been complaining to no avail about the inferior dimensions of the screen. Why are manufacturers still cramming them down our throats? This one isn't even a case where people prefer 16:9 to 16:10.)
The best form factor of any smartphone I've owned is the first one, the Nexus S: 63mm wide, 124mm tall, and the back popped off to make replacing the battery convenient.
Phones have gotten steadily worse, as far as usability goes, ever since. There's more computing power, but I have trouble believing that's what's driving the shape.