This is an area where we must acknowledge that one's "values" frame of reference is inextricably linked to how you decide this question.
Yes, we definitely are sacrificing some of everyone's bodily autonomy because we think it's less bad than the consequences of putting no limits on it.
Caffeine, Alcohol, Cigarettes, Cannabis, Cocaine, Meth, Heroin, Fentanyl, biological weapons
and on some parallel track:
BB gun, .22, Glock, AR-15, automatic rifle, RPG, cannon, tank, land mines, nuclear warhead
If there's even one thing on the above list that you'd not want to be automatically OK for every person in the world to own and operate (at least on a "until you hurt someone" basis), then you're putting an arbitrary limit which values "preventing likely harms to others" over absolute autonomy.
Even if you'd allow the warhead I don't think either of us can be said to be 'right' or 'wrong' -- just that it's most productive if we acknowledge which "paramount value" (perfect freedom vs. reduction in predictable harms to innocent others) we are prioritizing, and acknowledge that we have to make a significant sacrifice in the other one in order to do that.