There's a lot of "monster under the bed" psychology used by opponents of this stuff, same happened with gay marriage. It's an exploitation of information asymmetry basically, most people simply don't have time to really look into details of everything, so the default "safe" position is generally seen to be the status quo unless they personally experience something wrong (either themselves or via part of their close social network, family/friends/close coworkers etc). Dishonest campaigners can lie and posit endless "possible" harms that are hard to conclusively disprove so long as there are no real world experiments anywhere, and have proven to be able to quite successfully delay progress for a long time. Having it be illegal can be sticky as well since it encourages people to hide how they personally are connected.
However some consolation can be derived from it being a castle of sand and thus subject to a real tipping point effect. As soon as some trailblazer does manage to move forward and shine a light under the bed and it turns out that no, there is no monster, the sky does not in fact fall, and people realize that they do in fact know many people involved who were just hiding it before. Then it fairly rapidly becomes a non-issue to the majority of the population as they move on to new concrete (or invented, but new anyway) concerns. Hopefully the drug "war" will soon become the latest example.