Why would you sit quietly alone in a room when you could be enjoying a sauna, soaking up the sun at the beach, or picnicking in a park? I don’t think the problem is sitting alone with one’s thoughts - but rather that an indoor room isn’t a particularly desirable environment. At least it isn’t as undesirable as an airplane seat though.
When you're finally just thinking about yourself, your "desirable surroundings" will disappear. You'll be trapped in the discomfort of your mind. You'll then try to escape these thoughts by focusing on the picnic, sauna, beach.
Sitting in a room alone is just a metaphor for only thinking about your inner self. You can do that anywhere. And most people can't stand it, so they do something else. So, your conclusion is right, in that people would always prefer to do anything else other than focus on their inner thoughts. Like being in a place that distracts them.