Supply and demand. We're not running out of dirt and not running out of water (east of the mississippi) or running out of farm labor. So if your farmer neighbor works 7x10 and you work 6x8 you are going out of business.
Side job makes it better for individuals and worse for the community. My great uncle gentleman-farmed 40 or so acres of sweet corn and had a roadside stand on the major road he lived on. But the gas stations he owned paid the real bills. Meanwhile his neighbor corn farmers had to compete (on a small scale) with a guy who doesn't need to make money and thinks restoring tractors is a hobby (so he needs something for the tractors to do after he fixes them up, next thing you know he has 40 acres of corn planted...) Luckily his competitor neighbors had a lot more than 40 acres or he'd have put them out of business.
Any low barrier of entry, high supply low demand job is going to be very rough on the employees. Imagine being a supermarket cashier, which is bad enough, but also having to put up money to buy and stock the store before even being allowed to labor there. Meanwhile the guy down the street is doing it as a hobby off second mortgages or other employment to pay the bills. So that's farming.
The cost of food is high because of innumerable layers of middlemen, also lots of "value added" processing. Corn runs around 10 cents per liter (weird measurement system but whatever). Corn flakes cost a supermarket consumer a couple bucks per liter. The farmers get pennies on the dollar at most. That's why they're motivated to "do all that work for free" when selling direct to customers in CSAs or farmers markets. It must be heartbreaking to see three corn ears saran-wrapped at a food store for $1.50 and know as a farmer you only got a penny from that to pay all your bills, which probably near approach the penny earned. So selling ears at the "hipster farmers market" is a good deal at a quarter per ear. On the other hand if you "invested" four hours loading the trucks and driving and 8 hours sitting there, its easy to go broke on an amazing profit margin if it rains on the farm market that day so people stay home.
Anyway, aside from heartbreaking stories about family farms, we simply have too many farmers, so they're working for miserable pay under miserable conditions, mostly. Biggest scam ever by corporate PR is pushing the loss of the family farm story to keep the WAY too many middlemen story out of the news.