It's a completely reasonable assumption on his part because it is literally 100% guaranteed that society must fundamentally change due to fertility rates alone. Perhaps some think that fertility decline means populations will gradually decline on a scale of centuries or something? Which I agree would be generally just not that big a deal. But unfortunately that's not how it works. Fertility rates are exponential system that kick in hard once they start going.
It's easy to intuit this by thinking about society in terms of generations. Since a peak fertility window is around 20 years, a generation also tends to be around 20 years. So how much will population rates change as one generation dies, every ~20 years? It's easy to work out by example. Imagine a fertility rate of 1. That means each woman is having 1 child on average. You need to have 2 for replacement. It generalizes to a factor of fertility_rate/2, so with a fertility_rate of 1 the population will decline by 50% per generation, per 20 years. If you start with a generational population of 8, then you'd have child populations of 4, 2, 1. Once that 8 generation starts dying, the 4 generation will start dying about 20 years later. And when the 8 generation dies you lose 50% of your total population, and 50% again when the 4 generation dies, and so on every 20 years. That's going from 8 population to 1 in 60 years! This also emphasizes why immigration is obviously not a solution, the scale is simply far too large.
Also bear in the mind the vicious cycle these outcomes will create. With a growing population your economy also naturally grows, even if it stays (proportionally) the exact same size - because there's more people, more consumption, and so on. The exact opposite is true in cases of population decline. So you're going to be trying to encourage people to have more children at the same time that your economy is collapsing. You will also have totally screwed up age ratios - in our example your 60-80 year old group will make up 50% of your population, forever! So you also have a lower labor forces, extremely high costs in terms of social security/healthcare/etc, and so on.
So there is no way that any civilization can persist on any reasonable timeframe with a subreplacement fertility rate.