My big dig against Matrix is that it is monolithic. By compare, XMPP's virtue is in the very first word of it's name: Extensible (of Extensible Messaging and Presence Procotol; really: Protocols).
XMPP is designed to be grown, to have new capabilities added, to be experimented with. XMPP was used to build Google Chat's video conferencing a long long time ago (Jingle) because it was the only sensible straightforward chat system out there built for extensibility, built to be grown on. XMPP has all manners of neat uses, from IoT to VPNs (EdgeVPN) to nearly anything else you could imagine.
A lot of people seem to like the authority & control & centralization of systems like Matrix, but to me, this consigns away the future for an easy convenience. It restricts what a thing might become. I value loose coupling, I value extensibility highly, and XMPP continues to be an evolvable, changing, enhancing set of technologies that stays relevant because Extensible leads it's design. It's something Matrix will not nor ever could compete with, and to me, it nearly certainly dooms some latter version of Matrix to obsolescence: there will come a point when the bad choices have added up, and no progress can be made.
Let's take a little case study example. XMPP itself faces a moment of crisis now, imo, with Multi-User-Chat (MUC), which is a very not good experience & technically crufty; a very early specification that has really obstructed chat rooms from being good & let Slack et cetera take over & dominate. There are some attempts to fix MUC but the tech underpinnings are, imho, bad. The good news? There is a much wiser, much better engineered alternative, MIX, that builds upon many of the other XMPP standards that have emerged over the years & stood the test of time. MIX is a way simpler, way better, way smarter way to do chat in XMPP. The ugly? There are nearly no clients that have implemented MIX. Not a lot of xmpp servers have implemented MIX. Trying to drive adoption of this thing necessary for XMPP to survive has been hard going.
But still, to me, even with this struggle, even with these huge pains, & a non-adapting future, I still think this case study hugely shows the need for Extensible systems. This is just one realm, one battle being fought, amid a system that made up of a bunch of different components. Matrix might be able to muster up the need for serious breaking changes, but my feeling is that the concensus based, forward moving innovation of Matrix will keep adding decisions & technical nuance & capabilities forever, and will likely never have the chance for redress & re-appraisal that XMPP by it's nature, by it's namesake, promises. XMPP can re-orient, learn, adjust. Nothing prevents Matrix, with it's monolithic , core-contributor-driven specification system from making radical changes, but they will forever lack the outsider perspective & autonomy that folks can bring to XMPP specification development. Matrix can't be the platform for a next-Google to build their next-chat VR platform on, because Matrix is not extensible, it's not a system of protocols: it's monolithic, it's top down, it's structured for linear, forward progress, not the free innovation that extensible systems are designed for. Matrix has many decades of wonderful innovation & advancement to go, but to me, it's already a dead end, it's already made it's bed, where-as XMPP will always have a freer hand to keep becoming more, freer hand to becoming something different, is always available for radical innovation (such as EdgeVPN, or IoT).