These narrative simplifications end up just being confusing.
Neanderthals from 400kya are often classed as Heidelbergensis. These guys were less Neanderthal-ish and more similar to us... being closer to and less divergent from the sapiens-neanderthal LCA. Neanderthal-Denisovan divergence occurs at this time.. so calling them Neanderthals rather than Neanderthal ancestors is kind of messy.
There is a shortage of fossil evidence from this and earlier periods... It's called the "muddle in the middle."
In any case.... Sapiens also had ancestors at this time. We don't have fossils, but something has to be our ancestors. So if we are calling Neanderthal ancestors from this period Neanderthals... it would be more consistent to call sapien ancestors sapiens.
Individual populations may have been insular, small and most died out. But... there were people everywhere.
Humans existed over a vast range. From south Africa to Northern Eurasia. East to west. At this point in time... I think it's confusing to think of neanderthal/denisovan/sapiens as different species.
Individuals may have been inbred... but the overall genetic diversity across the whole range was greater than the genetic diversity we have today. In some sense, we are the inbred ones.
Also... population estimates are pretty dicey. We don't really know. Could have been booms and busts. Could have been ideal habitats with higher populations.
We still have a fairly poor grasp of human "natural history"