My own brain is idiosyncratic and doesn't always react as expected (e.g. to caffeine) and I've heard friends say that they think that taking marijuana during college might have permanently dulled the edge of their intelligence - which isn't exactly a controlled study or anything - but is enough to scare me away from ever trying the stuff.
And yes, I also never drink alcohol, and try to steer away from large doses of caffeine. Drugs scare me, legal or not.
Consider the risks and rewards very carefully before whacking on your brain with any sort of neuropharmeceutical - everything we have nowadays will be regarded by future times as the equivalent of a sledgehammer. The people trying to decriminalize and legalize drugs are, without ambiguity, the good guys. Don't confuse that with the proposition that drugs themselves are good.