The emphasis is on: done right, with the right people.
There is a reason why drugs are prohibited in many countries. People don't use them "correctly" and can become messed up pretty heavily.
I took shrooms when I was 18. After an initial phase of a lot of hardly controllable laughter, I experienced the strongest fear that I ever had in my life. Basically, the fear to remain stuck on this trip - nothing unusual by the way, as I figured out later by reading reports from other people. The main fear for me was to be stuck in this situation where your brain isn't filtering anything anymore. I was literally sitting around UNABLE to do anything. I mean, somehow I knew how to do a more complicated thing (like building another joint), but somehow I was incapable of performing the necessary steps. It was frightening, especially since there was no supervisor who could lead me out of this spiral of fear. The fear lasted only for 2 hours or so, then the trip was over as I fell asleep.
But I had panic attacks for another 6 months. Also a very typical time frame for many people. No fun. Stressful places, loud dogs, everything noise sort of made me feel anxious. On the other hand, it was a time where I thought about myself A LOT. I mean, really a lot. After 6 months, I somehow emerged with new self-confidence which grew a lot over time way beyond the self-confidence I had before (which could also be just a side-effect of getting older, I don't know).
So, now I wouldn't like to miss that experience. I don't know how it changed me, but I certainly do recommend that people should never try hallucinogenic drugs without supervision - people who can help you mentally to escape states of fear.