That is definitely true if you are farming on debt. The big time operator is happy to see no more than index fund-like returns, so he is quite content to dump everything else into interest payments. Which means you also have to in order to compete. I am less convinced by your friend if you are not trying to service debt with your labour input.
That said, you can leverage the debt. It is much harder to get someone to loan you money to invest in index funds. With the huge run-up in asset values over the past decade, I expect the farmer with a small holding 10 years ago, who then bought up everything he could with the bank's money, is now considerably further ahead than he who invested the same amount of money in index funds 10 years ago and took a job at Walmart to try and top up those investments.
But you are right that not all decisions are financial. I expect the Walmart job is far less stressful and that is worth quite a bit.