That's a pretty weak argument. We know that there are
certain situations in those sports that can cause brain damage, like getting hit on the head by a rock and lack of oxygen. We know them fairly well, and the majority of people avoid them successfully.
With rock climbing or driving or scuba diving, it's an "It might happen to you, but if you avoid these extreme situations it won't" situation whereas with taking drugs you can't really avoid pitfalls, other than taking care not to take too large a dose.
With drugs it's likelier to be "you get smaller amounts of damage if you take smaller doses" or plain "we don't know what the fuck this thing does".
I agree that it doesn't explain much about the attitudes. The default attitude should be ambivalent until we know more, not hostile.