While I feel prestige is a nebulous thing (it partly owes to marketing, optimizing for rankings etc), it is something that has real world implications in terms of an institution being attractive to talent etc. and talent, like interest, compounds.
With respect to Anglo-Saxon institutions, I don't think all of it just mere bias.
The talent pool issue is not a negligible one. English-speaking universities simply have a larger pool to draw from, and the best American universities in particular have the advantage of being able to not only draw talent from the largest English-speaking country in the world (itself, though arguably India too if we count L2 speakers), but also tons of aspiring students around the world who also speak English (especially from large countries like China and of course India) who become alumni, achieve success elsewhere and go on to boost their alma mater in their spheres of influence. Also the spreaders of international influence (the media) tend to be predominantly English-speaking, so Anglophone universities are naturally boosted over others.
Most non-English speaking elite universities are limited in that way -- Ecole Polytechnique for instance is very famous (and very elite) in France but has trouble drawing talent from the rest of the world. Not as many people are interested in learning French vs English...
The other is "forward momentum" I think, which has to do with research $ for producing work that matters. US schools have real money to hire and do "stuff that matters", at a scale that would be inconceivable at universities in smaller countries. It's also easier to assemble the right team with when you have a big pool of people to draw from both domestically and internationally, and to provide a strong academic community for folks to thrive in.
Also universities that optimize for selective admissions at the undergraduate level tend to have different goals (say, grooming an elite class for the country's public, private sector, etc.) from research universities, whose goals are to generate discoveries (the latter tends to be considered, for better or worse, more prestigious). As such, you may not be surprised to learn, most elite universities (at the undergrad level) don't have strong research programs.
There's a metagame here, and our instinct is to think that education rigor is necessary and sufficient to produce prestigious universities, but I think in the modern world, other factors dominate.