Maybe this is too reductionist but I once heard someone describe it as, "there's nothing fundamentally dangerous about pot, but one of its (often intended) effects is that is slows our engines down. And some of us need to be firing on all cylinders." School was a tricky time for me. I'm thriving intellectually now, but I was a C- student pretty much 4th-12th grade. I needed those cycles that pot would have robbed me…
South Park put it quite well: "Well, Stan, the truth is marijuana probably isn't gonna make you kill people, and it most likely isn't gonna fund terrorism, but… well, son, pot makes you feel fine with being bored. And it's when you're bored that you should be learning some new skill or discovering some new science or being creative. If you smoke pot you may grow up to find out that you aren't good at anything."
This ends up hurting our society on multiple fronts: people not being as productive as they could be because they're encouraged to waste time, an overall less happy society, a less healthy society because of the encouragement of sedentary behavior (and the economic burden that comes with that), a large part of the economy that could be devoted to improving people's lives but is instead wasted on encouraging negative behavior, etc. The problem is that we've grown so accustomed to this situation that we rarely think much about it, or what we can do to change it (whereas marijuana is getting at least some scrutiny, since legalization is new).