It's worth considering that our recent obsession with the "caveman diet" is the extension (and obvious continuation) of the early modern "noble savage" construct. It seems we ran out of culture clashes of colonization and now are trying to colonize the past to process our anxieties and dissatisfactions.
It's probably not that silly, if you consider this:
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB
A = How long Neanderthals existed (350 k years) B = How long modern agricultural societies have existed (7 k years)
Take into account earlier hominids, and the contrast is even starker.