I have slightly-conflicted opinions about this.
On one paw, I'm disappointed with how normalized, say, caffeine is, and how many people accidentally fall victim to caffeine dependence out of carelessness. In many cases, one's inability to function before their daily cup of coffee didn't exist before they got addicted. (Nowadays people at least know how terrible nicotine is, but most still think vaping it is OK for some reason... but I digress.) On the other paw, I'm also not exactly a supporter of the idea that "drugs are wrong", as you so kindly put it.
(Some people need caffeine to help an existing condition just like some people with ADHD might need stimulants, obviously I'm not talking about that but rather completely healthy people who develop a dependence on caffeine because they think it's completely safe and harmless.)
I believe that proper education about drugs would enable people to use them safely and responsibly—rather than the current situation where pervasive propaganda prevents many people from obtaining the education they would need to do so (also known as "harm reduction"). They can't tell the signal from the noise, so they have to shut everything out, even safety tips that would help them.
Don't get me wrong, not everything you hear about drugs is propaganda. For example, methamphetamine and MDMA are significantly neurotoxic and can wreck your brain and your life very easily and quickly. Heroin is so madly addictive that one often stops being of sound mind once they've taken even a single dose, if it's high enough.
But ayahuasca (DMT), or LSD? They are of course to be respected and used responsibly, but they are not "wrong" in and of themselves. Yes, there are ways to abuse them, and they are capable of facilitating severe mental damage and permanent impairment—but that doesn't mean they're simply wrong, even if they are risky.
You explicitly dismissed this but I want to say it anyway: set and setting matter a lot when you take a psychedelic. "set" represents your entire mental state. It consists of your mood, your hopes and dreams, your anxieties and traumas, your goals, and so on. People who get a bad trip were stung by set. Not because "drugs are wrong".
Psychedelics do not directly cause depression or misery or trauma—they give your brain the capability to create those experiences for itself. They create unrestrained thought and an experience that isn't subject to the usual moderation. Of course you can traumatize yourself on LSD—your barriers are down. You're vulnerable to your own thoughts. By taking LSD you are taking responsibility for what you do to yourself while you are on it, and what mental state you are in when the trip starts.
Drugs aren't wrong, they're just formidable.