It's comparing apples and oranges to compare medieval peasants life and modern office work. Even today in rural areas people work around the clock in and around the house to keep everything afloat and barely survive. Sitting on your butt for 10h a day and doing intellectual work seems hard only until you try to do some farming for a while - and modern farming is nothing compared to how hard it was back then, when you…
You don't go to sleep thinking about that big problem you still haven't solved. You actually get the satisfaction of doing the physical activity. You know with certainty that you've done your job (you can actually see it) and you know when to stop because your body will tell you to. Finally, you might be exhausted physically at the end of the day, but your mind is definitely not exhausted, which allows you to actually do something with it afterwards.
On the other hand with mental work, mostly you can't stop thinking about work unless you distract yourself. Many times you don't see the outcome of what you did and it's pretty easy to think you haven't actually "done" anything.
Another thing I'd say is that with farming (and other manual labor activities) there is very little that is not known beforehand. You might have some unexpected events, but even then, you likely know what to do in those occasions. With mental work it's a lot more unpredictable and full of unknowns.
Having said all that, I've never had the opportunity to work with physical labor so my views are likely biased.