No country views property rights as inviolable. Eminent domain trumps individual property rights. Property can be seized pursuant to a lien due to debt, non-payment of taxes, criminal activity, etc. We're not operating under some slavish, unyielding notion of property rights.
There's two distinct concepts being confused as one. The first concept is property rights in general. This concept is singular and correlated with the ideas of ownership and territory. This is may be a universal concept and may be a good candidate for designation as a "natural right". The second concept is the classification of what counts as property. This concept is numerous because we have to actually enumerate everything that counts as property.
Slavery is exceptional because it counts people as property. The error is not with the first concept of property but an error of the second kind, an error of misclassification. Owning a farm and working the land is ok. Owning a farm and owning slaves to work the same plot is immoral.
We see this reified during Reconstruction with Sherman's promise of forty acres and a mule. Where former slaves would be compensated by being granted property rights to the land they worked. Property rights were to be their salvation. As we all know that didn't happen. And a century of systematically denying the rights (including property rights) of their descendants (or anyone who looked like them) is the major reason for the disparities between black Americans and other racial groups.