Food can be addictive and destructive. Sex can be addictive and destructive. Money and power can be addictive and destructive.
"Recreational drugs" can be used in respectful ways. To many they are holy and sacred.
They can enhance your creativity, open your eyes to the wonder of the universe around you, and gift you with empathy for all living creatures.
With them you can experience death and rebirth and profoundly transform the way you understand and relate to the world. They can be intensely therapeutic.
There is certainly a stigma against all that, and against certain drugs and experiences which buck against the need for many people to reduce life to a little tiny and orderly box under their command where they can channel your world view and your understanding of yourself and others in a manner of their liking.
Yet despite all of their efforts they can not fully resist the deep need of humanity to transcend their ordinary state of consciousness. Each society has sanctioned ways of doing this, whether it's through certain approved substances (like alcohol), rituals, or ordeals.
Curiously, societies tend to differ as to which mind altering substances are prohibited and which are allowed. Usually it's the substances with a long tradition of use in the given society that are allowed, and the newer substances which are prohibited. The very same substances which are allowed and even praised in one society might be despised and prohibited in others (alcohol, mushrooms, and marijuana are some great examples).
Mind altering substances which a particular society is encountering for the first time and the use of which can not be easily fit in to the dominant culture's traditions are usually seen as a threat.
But this can change over time as the members of that society get more familiar with the substances and learn to use them in socially sanctioned ways.