The problem isn't if the population gradually shrinks, it's if there's an uncontrolled downward spiral in fertility. If the birthrate gets below a certain point, then most people won't have any experience whatsoever interacting with babies or young children in their day-to-day life, and cultural norms will shift to make childlessness the default option. This marginalizes people who choose to have children, which pushes the birthrate down even further. This has happened in South Korea, where children are barred from many public places and it's hard to find housing in urban areas if you have children because the noise they make will piss off the neighbors. The birthrate is currently ~0.7 births per woman, meaning that every 100 South Koreans will have around 12 grandchildren. Here's a good article if you're interested:
https://archive.ph/bM4FfA few quotes:
> Very little in Korean society seems to give young people the impression that child rearing might be rewarding or delightful. I met a stylish twentysomething news reporter at an airy, silent café in Seoul’s lively Itaewon district. “People hate kids here,” she told me. “They see kids and say, ‘Ugh.’ ” This ambient resentment finds an outlet in disdain for mothers. She said, “People call moms ‘bugs’ or ‘parasites.’ If your kids make a little noise, someone will glare at you.” She had recently vacationed in Rome, where adults drank at bars while their kids ran amok. She said, “Here, people would say, ‘What the hell are you doing?’
> An artist named Daum told me that, when he was young, “if you kicked a ball into someone else’s property, you went and rang the doorbell and got it back.” That city no longer existed: “Now you get yelled at—‘You could’ve broken my window!’ ” There’s a special word for noise between floors. Complaints forced Daum and his wife, Dani, to leave their previous building; one neighbor said, “I can’t stand your children anymore!”
> In the southern city of Gangjin, I stopped at a coffee shop and encountered a sign on the entrance that read “This is a no-kids zone. The child is not at fault. The problem is the parents who do not take care of the child.” The doors of Korean establishments are frequently emblazoned with such prohibitions. The only children I saw on Seoul’s public transit were foreigners. Kim Kyu-jin, who is by all accounts part of Korea’s first openly lesbian couple with a child, told me, “Five years ago, we didn’t think too deeply about ‘no-kids zones.’ Now we think it’s discriminatory. We always call places beforehand to ask if we can bring our daughter.”