> I wonder why Ubuntu/Canonical can never get their upgrade game to work properly.
Ubuntu developer here. Sounds like you have a really hacked up system there, and that it's your existing system that has issues. Under these circumstances, I don't think it's reasonable to expect an upgrade to be able to work smoothly. Scripts that handle upgrade paths necessarily need to make assumptions that what they are upgrading from is what the distribution put there. If you hack it up, then those scripts aren't going to work.
Your Arch system, being rolling release based, I assume isn't as hacked up with newer versions of things because you get them straight from your distribution? The equivalent on Ubuntu would be to run the six-monthly releases, which I assume you choose not to do. If you do that and then you hack up your system, then I don't think it's fair to compare to an Arch system where you (presumably) don't do that.
If you want to run an Ubuntu LTS system then I suggest that you do your hacking inside containers (whether system containers like lxd, or app containers like Docker). Then you won't need to hack up your system itself, and then upgrades will generally work fine.