This short essay is not pg at his best in my view. Spending time having fun is Plain Good, and necessary, even for the most focused startup founder. But watching TV don't count obviously. Playing cards with friends at a terasse does.
There are many positive activities that can be both productive and fun, including a card game with friends. In that case, you are investing in to those relationships and fulfilling a very basic human desire for socialization. It's not that you're not getting anything done. You're just getting different things done. Compare that to a day in front of the TV. Can you say you've gotten anything out of it? Did you learn a…
I think your point of view, and pg's, is very utilitarian. Utilitarianism has its uses in many human activities, including writing code, but should not be invoked when talking about playing cards with friends. When I play card, I do NOT "invest" in relationships, I have no intent to fulfill anything. I am just having a great time with friends. Playing cards, or fishing, or chatting idly, is just a way to spend time doing nothing. And doing nothing, ie spending time without any "investment intent", is the way to enjoy fully your time.
Take another case: if you spend time reading a lengthy novel, say Proust, what is it for? Do you read it because you want to tell your friends you read it? Do you read it because you think it is an "investment", adding up to your knowledge of world literature? Do you read it because you want to learn how to write a novel? Learn about Proust? Understand why so many people admire this book? NONE of these reason is valid. All of them are "intents", "investments". All of them will actually destroy in the egg the only valid outcome of reading this novel, which is pure reading pleasure. (You'll have to pass some barriers to feel this pleasure, in the beginning)
The same holds for painting, music, etc. If you go to a Concert hall listening to Mozart or go to the Louvre and take this activity as an investment in your knowledge of art and your ability to talk cleverly about it, you better stay home watch TV.
The flip-side of this is that by actually reading lengthy novel, playing cards idly with friends, etc., and doing so without intent, you will in fact learn a lot about life, about humans, and also how to handle relationships. But the "without intent" part is essential.
To take your example about the president: I would LOVE the presidents of big or small countries to spend a lot of time doing nothing, chatting with "non-politician" friends, read lengthy novels. They would probably do less mistakes.
We live in a world where we are made believe that every second counts, like in TV series. It is not true. The only reason to live on earth is to be happy, happiness is a state of mind that need some idle time to spawn, let's stop running behind our ghost's shade.
[Edit: typo, clarity]