For millennia, there was no growth because available energy was strictly limited. GDP of a country was a function of its population, itself strongly related to its agricultural capacity.
Read ancient history; you'll see that empires, for ages, relied entirely upon pillaging their neighbours for their own enhancement. The world economy was a strictly zero-sum game basically forever.
The political constructions of the past reflected that state of the matter. There was no social mobility because resource scarcity commanded that as a son of a mason, you couldn't be anything but a mason. And of course, 90% of people had to work in agriculture, anyway, just to feed everyone. There was no way out. Of course society was driven through rigid hierarchies.
Then at some point some guy discovered America, and a strong, growing influx of riches began pouring into Europe. People discovered absolute, continued growth. Then as this trend continued and England had cut down all its trees, they started using coal, and the movement accelerated itself, and went on until now.
As Jancovici remarks, we always went from some energy source to a more powerful, denser, more reliable one: hunting to agriculture, windmills and hand pumps to coal, coal to oil. We never went the other way around, but at some time we'll have to. How to cope?
Now let's talk for a minute about GDP. I had quite heated argument with people with economics background. Basically, they say that GDP can grow eternally because GDP is related to value, which has no physical existence. So though GDP generally implies some form of materiality (be it in the form of ore, grain, industrial goods, or services) it's not mandatory because you could for instance build a better car using no more materials than the previous one.
However it seems to me that this quickly wanders into philosophical territory: what is value? In economics, it seems related to human satisfaction. As the number of human beings on earth is physically limited (we could be 10 billions, 100 billions, but not an infinity), at some point that implies that individual satisfaction can increase indefinitely. That's a stretch, however you see it. That would mean that two humans exchanging a given set of resources in a closed room could increase indefinitely the satisfaction they get from their relationship, and so any arbitrary small country with as few inhabitants as you want could have an infinitely growing GDP, and reach individual nirvana and beyond. BTW I fail to see the point of free trade at this point... Standard neoclassic economics are full of such logical holes.