Off the top of my head, there's Jared Diamond's "The Worst Mistake in the History of the Human Race" [1] (popular writing, so grain of salt) and James Scott's more academic "The Art of Not Being Governed" [2]. From the first reference, "One straight forward example of what paleopathologists have learned from skeletons concerns historical changes in height. Skeletons from Greece and Turkey show that the average height of hunger-gatherers toward the end of the ice ages was a generous 5'9" for men, 5'5" for women. With the adoption of agriculture, height crashed, and by 3000 B. C. had reached a low of only 5'3" for men, 5' for women. By classical times heights were very slowly on the rise again, but modern Greeks and Turks have still not regained the average height of their distant ancestors."
So a few points. First of all, you elide between comparing hunter gatherer societies to early sedentary societies and contemporary industrialized societies. They are very much not the same, and I prefer contemporary industrial societies over both. This was purely a statement of leisure time, which was higher in HG societies than contemporary ones.
The comparison between early sedentary and hunter gatherer societies is where hunter gatherer societies have a pretty attractive value proposition. Some points: the threat models between agricultural and hunter gatherer societies are different, and you can sustain a higher population per acre with agriculture, obviously. But people never wholeheartedly embraced sedentary, centralized societies. They germinated in particularly fertile areas where states could most effectively maintain themselves and extract excess value from highly productive land and labor.
But if being agricultural and sedentary were an obvious, natural course of events that everyone would choose, you'd see agriculture and sedentary states rapidly spreading to the boundaries of the geography that can support people via agriculture. That's not what you see: you see constantly fluctuating exteriors in a contested relation with borderlands. The people living in those areas constantly switched sides depending on convenience and what was best for them at a given time. For instance, sometimes, Han China would offer incentives for people to take up sedentary agriculture (no taxes!), and they did. But this wasn't stable, and people would be more than happy to switch back to other lifestyles depending on the state of the economy and the incentives they faced.
Over time, the total area of those borderlands shrunk as a result of technological progress and more capable states. But this took literally millenia.
[1] http://www.ditext.com/diamond/mistake.html
[2] https://en.wikipedia.org/wiki/The_Art_of_Not_Being_Governed