As I said, I'm not a libertarian. I'm fine with fair and reasonable taxes, you pay those in return for something else, but I'm generally against approaching it the other way round, as in "you might be allowed to keep some of the fruits of your labor, but they actually belong to the state/tribe/party/volk".
You're not getting "all your wealth", the market prices in a difference in what you make and what it's worth for others. But generally in that line of thinking, you could just as easily ask why you should be entitled to any of your wealth, since you're not in a vacuum and you wouldn't have any of it if you weren't part of the system.
I think it's primarily a question of whether you consider the individual as an agent or a cog in the agent that is society. I prefer the idea of individual agency, with all the issues it carries.
> rather than starting from self-ownership
Without being free, you cannot agree to any rules. And you can also not be allowed to leave if you don't want to live in that specific system of social cooperation.
> It seems equally silly to me to structure society so that just because you happened to be born into poverty, your life choices are substantially constrained due to no choice of your own.
Oh, sure, the luck of the draw is a major issue. I don't think making everyone a slave is the answer though. And I do believe there's a good reason to organize society so nobody is born into poverty: it's more efficient. Having millions of people sit around and not contribute is a waste, both of their time, energy and mind, and of the resources they need. I don't think just increasingly overtaxing those that do contribute is the answer though.