I think you’re being unnecessarily pedantic. Sure, there are caffeine and marijuana and alcohol users everywhere, but there’s a huge difference between your neighbors quietly having a drink in their home after work compared to loud alcoholics getting in fights and stumbling around on the streets.
When someone says there is a lot of drug activity in a neighborhood, they’re not saying that people are quietly consuming an edible in their basement every once in a while. Problematic drug use seeps into many aspects of social life.
And before anyone assumes that I’m referring to poor neighborhoods, I’m not. One of the most problematic and dangerous drug environments I ever witnessed (in my relatively sheltered life) involved some very well-off young people who got caught up in drugs, then dealing drugs, and then severe addiction that interferes with their judgment to the point that they were getting violent and threatening murder. This was in a well-off community where a number of families ultimately chose to move away to escape the parts of the community with a drug problem and had no regrets for doing so.
I’ve also crossed less dramatic situations where social groups went downhill due to less overt drug use: People who start using unnecessarily high non-therapeutic doses of Adderall to become energized (doesn’t last, causes weird problems over time), people who start using benzos with casual disregard (turns into a dependency very quickly), and even people who casually use opioids for recreation as if it’s a sustainable activity. In each case, the effects of excessive drug use start taking a toll on the person that extends to the social group. Getting away can be like a breath of fresh air.