As a person who recently wore contacts every day until dry eyes and an injury forced me to switch back to glasses, I have almost entirely the opposite experience. It bothers me to wear glasses again. My prescription is quite strong, so it creates a pretty significant distortion, stronger towards the edges of the lenses. It's hard to do sports, as I now have a background fear about losing/breaking them, and the distortion messes with my hand/eye coordination. I had an episode skiing where I actually got nauseous from disorientation. As a result I do far less sports and that leads to other things. Ultimately, it has kind of screwed up my sense of scale of things, too, because the glasses make everything appear a bit smaller.
When I take off my glasses the distortion goes away, of course, but I can't read street signs, see people's expressions, watch movies, and other things. Watching an incredible sunset or mountain scene with my own eyes...it's like looking at a 200x100 pixel image Gaussian blurred to fill the whole field of view.
I might agree with you if my prescription is less, but TBH I wish I could get rid of these damn spectacles.
On the other hand, it did teach me a bit about philosophy. E.g. now I realize that to see the world "clearly" I require lenses, I just never noticed before. All of my perception is actually through lenses, not just my eyes. Touch, taste, smell, feeling, all distorted by my senses. And it feeds into my mind, the biggest, most distorted lens of all :)