You received six answers along the lines of "drugs do long term damage."
That may be part of the answer, but it feels incomplete. Elite training itself is often unhealthy. And doping techniques vary wildly. There are risks to HGH, EPO, anabolic steroids, and blood doping, but they have radically different risk profiles.
I think it's more that people have an idealized notion of athletes accomplishing something through sheer force of will. So... a scientifically designed training regimen seems fine, chemical alterations to your blood chemistry do not. Training at high altitudes is fine, replacing your blood with blood from when you were at high altitudes is not. Overtraining, eating disorders, destroyed tendons, encephalopathy -- none are ideal, but we seem to tolerate them.
Questions about what to tolerate are fuzzy and pervasive. Testosterone levels are monitored in women's competitions and someone has to decide the cutoff. Progressively sleeker swimsuits were fine, until they weren't. Equipment manufacturers generally advertise increased performance, but if they deliver too much too soon, they get banned.
These are all versions of the question, "What is this sport?" It's deceptive. Seems simple, but after examining all the edge cases, it becomes impossible to deduce from pure reason. Ruling bodies just pile on conventions.
A Supreme Court dissent once argued:
"[S]ince it is the very nature of a game to have no object except amusement (that is what distinguishes games from productive activity), it is quite impossible to say that any of a game’s arbitrary rules is ‘essential.’ Eighteen-hole golf courses, 10-foot-high basketball hoops, 90-foot baselines, 100-yard football fields – all are arbitrary and none is essential. The only support for any of them is tradition and (in more modern times) insistence by what has come to be regarded as the ruling body of the sport[.]"
Banning some supplements but not others is perfectly reasonable, but the fine lines are going to be somewhat arbitrary.