There's a certain irony in people thinking this was a bad decision. Let me explain:
The right against self-incrimination originally arose because of a quirk in English law relative to continental European law, and middle-age French law in particular. See, in France judicial execution was only permissible if the accused confessed. You couldn't use evidence as justification to execute someone; the defendant had to confess. This was considered humane and progressive--evidence might be false, but surely nobody would confess to a capital crime they didn't commit.
But this logic led down a terrible path. You could have the most brutal murderer in your hands, whom everybody knows without a doubt was guilty, but unless they confessed you could never execute them. (Life in prison just wasn't a thing because the state didn't have such an apparatus, at least not for common criminals.) So in a cruel twist of logic, there developed the system of torture for extracting confessions. Torture couldn't commence without eye witnesses, but this type of evidence wasn't taken in a proper trial (certainly not like we have today, or even as the English had at the time), AFAIU. So ultimately what you had in France and some other European countries was the most brutal criminal system imaginable, all because they were too absolutist and ideological in their understanding of how evidence can or should be used to mete out justice in light of the risk of error.
By contrast, what developed in England was a much different framework. In England any probative evidence could be used as proof of a crime, even a capital crime, so long as it convinced a jury, and so long as the defendant could likewise introduce any probative evidence that could exculpate himself. The English thought the French system of torture and extracted confessions abhorrent (just as the French, ironically, thought it unthinkable the English could execute someone based on circumstantial evidence alone), though that abhorrence was slow to become comprehensively enshrined directly in the constitutional law, thus the notorious instances and regimes of torture and extracted confessions in England. But those instances don't detract from the overall weight and theoretical foundation of the law.
Anyhow, my point is that in the spirit of English Common Law, the foundational rule is that any probative evidence should be admissible. Silence can certainly count as probative. And the core constitutional principle isn't that self-incrimination, per se, is bad, but that reliance on it can incentivize inhumane treatment of people. Furthermore, Miranda Rights are like a secondary or tertiary safeguard in service of preserving the core principles. All of which is to say that, while I'm not sure I would have decided the same way as those conservative judges, and to the extent we presume their sincerity, I can certainly appreciate the reluctance to exclude probative evidence based on an overly rigid and absolutist conception of how the government can make its case. Because history has proven that you can easily end up with the precise, extreme consequences--an unfathomably inhumane system--you're trying to avoid. The societal need to prosecute criminals will never go away, so there's always a balancing act at play. The lesson the English system took to heart is that, all things being equal, the more relevant facts you permit, the better.
EDIT: Much of the above was said more succinctly by the 15th century Chief Justice John Fortescue in De Laudibus Legum Angliae:
> For this reason, the Laws of France, in capital cases, do not think it enough to convict the accused by evidence, lest the innocent should thereby be condemned; they choose rather to put the accused themselves to the Rack, till they confess their guilt, than rely entirely on the deposition of witnesses.... By which over cautious, and inhuman stretch of policy, the suspected, as well as the really guilty, are, in that kingdom, tortured so many ways, as is too tedious and bad for description.