Have we forgotten the Linux Code of Conduct controversy? Contributors threatened to remove their code from the kernel. A CLA would completely prevent that.
No. Random fringe anonymous emails from non-contributors claimed (incorrectly) that developers could do so.
See https://sfconservancy.org/news/2018/sep/26/GPLv2-irrevocabil... and https://copyleft.org/guide/comprehensive-gpl-guidech8.html#x... for details (incorporating actual legal analysis).