As someone with 10 years drug experience, I'll agree with you.
I often see people say LSD is provably safe, when we don't even understand the brain.
I'd say instead that it's empirically proven that some people have suffered from negative psychological side-effects for months after an experience. You just have to read boards. From what I've read, it can last up to 6 months.
A female friend of mine actually had panic attacks for a while after. It's rare but it happens. And from my own experience, I'd say it has changed me probably more than anything else I tried, in ways I can't describe. I'd rather see someone responsible take ecstasy once or once a year, even though it's addictive and potentially neurotoxic, especially without antioxidants, it tends to be a spectacularly positive experience, emotionally and psychologically, a lifetime event. Also there are studies about how it may help against neurosis, shyness and all. And doing it with close friends or a lover, will give memories to cherish. At least it did for me. It's dangerously addictive and very negatively connotated, it doesn't have the esoterically intellectual marketing appeal of LSD, but I'd dare say it's more intellectually interesting, in that its rewards are clear, human and real.
There are a lot of people who take LSD regularly, so it's not that risky, but it's not trivial, especially for intellectual people with anxiety, higher sensitivity, and often difficult lives and childhood. But it's certainly interesting and perspective-shifting.