Can we have prosperity without growth? No. Can we have sustainability with growth? No. Then we can not have prosperity and sustainability. Can we afford not having sustainability? No. Then we can not afford prosperity.
It's a triangle: prosperity, sustainability, population. Choose 2 (and obviously if you don't choose sustainability the sh*t is going to hit the fan at some point anyway).
The key issue is the global population.
(Edit: We're discussing life on this planet. Colonising space obviously means that human population may continue to grow, but not on Earth and not using Earth' resources)