> We would have discovered poisonous or second order effects long ago.
Here's an easy counterpoint to that:
Which mushrooms are poisonous?
The answer is: We don't really know.
We know a tiny subset that we know we have not been able to implicate in any harmful effects. So far.
We also know a number of mushrooms that are lethal uncooked and unless cooked in certain ways (commonly you have to boil off the poisons) and where poisons are known or suspected to accumulate in the body and where people are thus recommended to avoid more than a meal or two per season with the expectation that this will keep your total exposure to below dangerous levels over your lifetime. But most of these have been discovered because they have toxins that are strong enough to kill or cause serious illness if the mushroom is not treated correctly. An example is Gyromitra esculenta, which is frequently implicated in cases where people who didn't bother to read up on it before preparing have gotten sick after standing over the pot while boiling it, or even just keeping windows closed.
But toxins in mushrooms are incredibly tricky. Many known poisonous mushrooms first affect people after weeks, at which point if you're unlucky, some vital organ just shuts down and it's already too late. In the case of many of the known poisonous mushrooms, we don't know how they work, or are not even sure exactly which substance in the mushroom is dangerous. The connection between the mushrooms and the negative effects are in many cases a "recent" (as in last hundred years") discovery as we've gotten more sophisticated about identifying the causes.
Given the nature of a lot of mushroom toxins we do know about, we have little to no basis for saying that none of the mushrooms that are commonly eaten have substantial long term negative health effects. (Yet they are too yummy to resist..)
In the vast majority of mushrooms, we simply don't know if they are dangerous or not, at all, or even if they are used as food, as there is a lot of poor knowledge of mushrooms amongst people who do in fact pick and eat them, and mushroom species are frequently confused by pickers, which both means a lot of the knowledge we have about them are dicy.
This is intentionally an extreme example since you made an overly broad point.
It is probably true that for a lot of "natural foods" we can say with reasonable certainty that they are safe. But the history of identifying dangerous mushrooms have amply demonstrated that people have trouble connecting cause and effect in terms of diet when the events are separated by even a few days, and that even scientific investigations into it can take a lot of time to determine harmful effect even when caused by - in some cases - powerful neurotoxins.
There's really no reason to assume that the long history of natural foods would lead people to realise harmful effects from it that are separated from ingestion by all that much more than that, so apart from the really immediate effects you can pretty much ditch most experience older than a couple of hundred years as irrelevant in proving the safety of foods. Of course that still leaves a substantial amount of time and scientific inquiry into natural foods that gives it an edge in claiming safety over something completely new, but the scales are nowhere near as clearly tilted in on direction as you make out
EDIT: Another interesting thing to look at, is how the knowledge of how to prevent scurvy was long limited to the extent that it was a military advantage for the British for a long time, and how they subsequently lost the knowledge: http://www.thenakedscientists.com/HTML/articles/article/forg...