Hardwood in the USA is mostly farm grown, so I'm not concerned.
I'm far from an expert, but I am in the process of planning our hardwood harvest in Vermont. The northeastern US produces a lot of hardwood and my impression is that very little of it is "farmed".
Can you point to sources? My attempt at searching didn't produce much either way. If you are right, I'd love to update my false impression. Or are you possibly confusing "hardwood" with "softwood", such as pine?