One of the big problems is that folks judge Matrix based on the legacy Element apps, which have now been succeeded by Element X:
https://element.io/blog/we-have-lift-off-element-x-call-and-... etc. Element X kicks ass, in my (very biased) opinion: it's a super-speedy Rust core with fancy SwiftUI and Compose native UI layered on top. It radically outperforms any other encrypted messenger i've used in terms of UI perf and usability.
However, because it's a rewrite, it doesn't quite have feature parity with the old apps, which are now over 10 years old: Threads is in beta; Spaces haven't landed yet, and Widgets aren't implemented yet. Therefore, we have to keep the old app around for users/customers who depend on those.
As a result, >80% of the people who say "Matrix sucks" are actually talking about bad experiences on the old Element mobile apps - rather than better client Element X or indeed Matrix clients from other folks.
There's also a large set of people who got bitten by encryption problems, almost all of which were fixed by Sept 2024.
Finally, there's folks who got bitten by the sad history of bridging in Matrix: IRC bridging used to be relatively okay; the team then got very stretched due to lack of funding; we tried to land a major PR to improve its architecture; the PR introduced bugs; Libera got very upset; we tried to fix things but failed to do fast enough. As a result, bridging to Libera in particular is awful these days, using adhoc bridges which funnel all traffic through a single user, with no ability to join arbitrary IRC channels on demand or use Matrix as a bouncer.
These days, the priority at Element is providing a self-hosted, decentralised WhatsApp and Teams replacement for governments... and once we get sustainable doing that, we'll be able to spend time building community features once again.