If you lived on a desert island by yourself, the only thing you would get is what you get by hard work. Your life would be better very directly by your work.
If you had another person on the island, you could share and exchange, and if you both of you worked, you'd both be better off.
By induction, adding more people can result in more people being better off, especially when you add specialization, etc.
Where that breaks down is when others start using force to take your/others stuff. Then you don't see the product of your labors, since someone else is stealing it.
Exploiting colonies fits that, so does the modern warfare/welfare state, the late Roman empire, and communism/socialism. _Anywhere_ that force is used to "reallocate" the products of someone's labour, the people start to get the feeling that working more won't get them ahead, because someone else will take what they produce.