I have also often complained about this pain of CI systems.
One of the problems is that the things you need to have a proper CI system conflict with fast build times. Proper CI requires a clean checkout, and a total compile fromscratch. If you're doing something in Docker, you ought to start your Docker process from scratch. If you're in a VM, you really ought to revert to a snapshot to make sure you're not accidentally accumulating un-CI'ed state. And so on.
While in normal development you may have a very fast turnaround, a properly configured CI system needs to assume the worst, start from scratch, and build everything, in every combination you support. (You may also want a less accurate CI build that trades speed for accuracy and just does an incremental build of some particular aspect of the system. But that should be supported by the full CI I describe here.)
Consequently, something that fails only 97% into that build process, and only fails on the CI server, can be very annoying to fix. But you don't really have a choice, because any hacky alternative is too risky. A CI system that has human intervention isn't a CI system.