Romans didn't just pillage, they also planted stuff like chestnut trees as they moved up and down Europe, so that they could use them in following years. Regardless, I find it difficult to believe that already-struggling tribes would further starve themselves just in case this or that opponent tried to invade. It's much more likely that they simply weren't able to grow much more in what was a hard soil, without the stronger plows that would be used later on to break such soil more effectively; especially when a lot of these people were still basically nomadic.
Production beyond the Rhine simply was not great even in times of peace, before the middle ages. Rome ended up controlling most of the German territory through client states anyway, even sending troops for punitive raids and propping up this or that friendly ruler, so they knew the economic potential of those lands pretty well; they just renounced full invasion because it was not worth the risk. At a time when they were already hitting what we could call scaling limits in their ability to mobilise troops over long distances, there was little appetite for going further North, where clearly there were no riches waiting for them. A similar assessment was done for Scotland, and rightly so.