While physical activity is good, high level competitive sports takes a toll on the body. Adding doping into the mix means that you willingly encourage athletes to take even more risks with their health and life. You also create a much bigger opportunity for countries with loose human rights track records to abuse individual athletes they have control over, as having Olympic medalists is used as a political tool for politicians to improve their standing on the global stage, and to improve their image within their country.
You should read up on the history of doping, it's not pretty. An example I have in mind is female swimmers from USSR who were forced to get pregnant (by their coach) and abort before the competition to rise their hormone levels, as it was thought to help their performance. Compare images of what female swimmers looked at the time vs what their look like now.
I don't think anyone who cares about human rights want to go back to that era. And