My recollection is that pairing Swiss tournaments manually actually got easier as the tournament got larger.
More players did make the mechanics of manipulating the index cards harder: splitting into score groups, sorting each score group by rating, splitting each group in half and matching to make the preliminary pairings all would be more work the more players you had but those steps are all routine steps.
It was the next part, where you adjust the preliminary pairings to ensure things like not having anyone playing someone they already played (a requirement you must meet), having people play someone in their score group, alternating colors, not playing the same color three times in a row if you cannot give them alternating colors, and whatever other ones I've forgotten about where having a lot of players made it easier.
When you've got lots of players you'll have bigger score groups. If you have to shuffle some players in the pairings to get things to work out you've got a lot more room with a big score group to do such shuffles while still keeping the players involved playing in the right score group and close to the same position in their respective halves of the rating-sorted score group.
When the score groups are small you've got a lot less room to move within the score group to shuffle people.
The net result was that a larger tournament made the easy parts of pairings harder but made the hard parts easier so was a net win.