Substitute "tenement apartment" for "shack" and "slumlord" for "former truck farmer", then subtract "living near formerly productive fruit trees". You now have a story that describes many times more people than your original.
The politics of privilege basically says that unless you have had the worst life that it is possible to have, you have to yield to anyone that had anything worse off than you. Since the word "elitism" is what brought all that into the conversation, perhaps it was a poor choice.
While at previous times in history, it might have been a mark of elite privilege to not have to pick your own fresh produce, that is no longer the case. Now, having fruit trees and a garden in your backyard means you are lucky enough to have direct control over a portion of your food supply, and can therefore choose to grow food with flavor and nutrition, rather than superior shelf life, visual appeal, and transportability--which are the characteristics that the grocery store will choose for you.
The apples that grew in my (rented) backyard as a kid were sour, with a woody texture, and were usually covered in wasps or ants. But they certainly did taste better than store-bought Red Delicious. The ants added a lemony flavor. I never thought to try the wasps.