Growth is an international arms race.
I think the biggest problem is that economic growth gives countries a (military) advantage. If you opt out of growth-based economic system, your neighbors will have more army or resources or manpower. They will eventually invade you or dominate you economically.
This is similar to the situation Africans were during Industrial Revolution. Europeans needed a lot of workers for factories and plantations, so they would visit a coastal African country and buy slaves. You could refuse to sell them slaves, but then they would sail to your neighbors and sell them firearms. Then those neighbors would have substantial military advantage and could invade you. There were only two options: be the ones that receive the rifles, or form a large alliance which refuses to sell slaves to Europeans. Forming large alliances is HARD. Africans weren't stupid - they tried to negate the technical advantage, for example the king of Kongo wanted to pay well for ship technology.
(source: "Black Mother" by Basil Davidson)
Back to prosperity without growth. To resist pressure from neighboring states that practice economic growth, you would need to continuously improve efficiency and that's hard. Produce terminator-like killer machines and drones to make up for less numerous army. Have super advanced science, strategy or diplomacy. The only country I know that comes close is Switzerland, and it has the advantage of mountainous terrain away from major transport routes. Ideally the advantages you have should be ones that can't be taken away from you or "stolen" via espionage.