Earlier quoted context omitted.
In Bilbo's youth, the Fell Winter took place, and "White Wolves" invaded Eriador. The Great Plague some time before and the Long Winter are explicitly said to have depopulated Eriador and left it "desolate." The Rangers were also not infallible. Obviously they did nothing to stop the Nazgul or large numbers of Saruman's agents in the time of the Lord of the Rings, nor did they stop the Orc-band that invaded the Shire…
See though that all of these disasters are spread over 1500 years. Most of them supposedly kill people, but they don't make the land uninhabitable and nothing stops from next generations repopulating them. Keep in mind that Europe suffers at least two big plagues, migration waves, a mini ice age and lots of other disasters. The main depopulation factors are hunger and food logistics collapse.
If the population density was low to start with, and food production tough, recovery from even a single major enough event can take a very long time.