Considering generations in the past, and still in a lot of poor countries have raised multiple children in the smallest of spaces, I just don't believe there is a causative relation. Much more plausible to me is the association to women's education, causing women to delay their first pregnancy and opt out of further children. Also there seems to be an association between general misery or fatality and birth rates.
It is not really a happy life if you have to spend your evenings in utter silence because there is a baby or a toddler sleeping in the same room two meters away.