These are the reasons:
1) They are too wide. Real farm tracks are often narrow and overgrown. You want a narrow viechle to fit down them.
2) They are too heavy. When they get stuck you want them to be light so it's easy to pull them out. Ideally a little manual pushing should be enough
3) Off-road "capability" isn't actually that important because you could get a Toyota Camry down 99% of farm tracks. The 1% that's impassible by a Camry is also impassable by most off-road viechles because a pipe broke or a river flooded and washed away the road completely. You want off-road capability to prolong the life of the viechle, not to get down the track.
4) You need good visibility from the cab so that you don't run over your colleagues when doing complicated maneuvers.
Those little Japanese pick-ups look perfect.