This has been my experience as well. I, too, have met and talked to thousands of people from every possible background on seven different continents of this world. From the poorest people on Earth to some of the richest. I've seen the same thing. About 5% are nasty and about 1% are truly bad people who commit horrible crimes.
Much earlier in life I helped recruit for a cult (much has changed since then). 3% - 5% seems on the high side for people who "just suck" -- perhaps by an order of magnitude. But perhaps the "just sucks" is context dependent. I found early on the FORD bulletpoints makes it pretty simple to start smalltalk (Family, Occupation, Recreation, Dreams/Desires).