Size has nothing to do with size or modularity. The same Qualcom chips get used from smart watches to tablets to cheap PCs, there's plenty of variety going on there. The difference is that the mobile SoCs are designed differently, requiring a complex system of bootloaders to even turn on that can change between models or even revisions of the chips.
The second problem is the proprietary nature of the chips. The PC became popular because other manufacturers copied and reversed IBM's BIOS and hardware, bringing a huge variety of available hardware that runs the same software. The only way that model could be profitable was to make sure everyone stuck mostly to the same standard, because breaking compatibility made your computer significantly incompatible with existing software your customers want. The crux of this requirement was that the user brought the software to the hardware, and the manufacturer could at most provide an operating system with a few tweaks at risk of breaking whatever the user wanted to run. This backwards compatibility still exists today, to the point that many BIOS update tools from manufacturers will boot FreeDOS to flash chips, running a software architecture older than many of their users on bleeding edge x64 chips.
The mobile ecosystem is designed so that devices last two, maybe three years, after that consumers throw their phones out and buy a new one. If you want to use your phone for longer then good luck, hope you bought a brand that ships you updates.
Qualcom, Huawei and Mediatek do not share the same level of compatibility the original PCs did, because they control the software supply chain. You don't take your software to Qualcom and make it work, you take Qualcom's kernel and then write your software. If Qualcom doesn't want to update the kernel version for your chip, good luck getting Android to boot; their open source drivers probably won't work because they either don't exist or are shoddily designed to fit only one kernel.
There's ongoing work over at the postmarketOS to mainline existing devices. Mainlined devices can run a proper, modern version of Linux with normal update support like any operating system, though Linux distributions often fail to properly implement things like deep sleep that are essential to using a phone. It'll be a while before mobile Linux becomes viable.
Then there's Apple, kind of a weird manufacturer in that they provide updates for well over five years even for slow mobile devices. They can do this because they don't need to buy chips from a manufacturer with an incentive to provide as little support as possible so end users buy new devices. They're also too popular for Linux not to hack in support for their weird, proprietary architectures.
All modular prototypes for Android have had the same problems with chipset and operating system constraints their constant-hardware cousins had; the same kernel supply line, with the same update problems.