You are wrong. If you're buying a house it might have taxes attached but not side lots if you read the site I referenced. The city is literally giving side lots away so that they are added back to the tax rolls.
Also absolutely no one is going to buy a house with $50,000 of taxes attached! Instead you wait until the land bank owns it and either buy it directly from them or wait for the yearly auction.
Rarely do housing sites have chemical contamination. There are however former industrial sites that do. But if you look at who is doing urban farming in Detroit it is exclusively on former residential land.
Urban farming does make sense economically. Lots of young people don't have the capital to buy good rural farmland. If you're in the city you're also near great markets but the key is growing high value, labor intensive crops like fruits and vegetables.
Unless you're an experienced farmer operating at scale you can almost always buy corn, wheat and soybeans cheaper from the farmer than you can grow them yourself.