What you're saying is true but only in a society where you simply allow all drugs. Ideologically I am of this inclining since I don't think the government has a right to determine what people should or should not be able to put in their own body, but at the same time I also think this is a path that is detrimental for society. A few factors you don't hit on:
- many drugs are addictive and this can lead to addicts turning to crime to try to fuel their addiction. yes we could offer more public support for such things, but these tend to be lifestyle choices that people revert to over and over.
- some drugs themselves can have a strong connection with abnormally violent and dangerous behavior - for instance, PCP.
- other drugs can have potentially extreme side effects even with minimal usage. olney's lesions / brain lesions are but one example. of course the same is true of e.g. alcohol, but the quantities involved are quite different.
- any drug that is illegal will result in a black market. this black market tends to trend towards violence. there is potentially big money to be made in drugs and criminal enterprises tend to attract criminals who don't just constrain their violation to their product.