Earlier quoted context omitted.
In an OS kernel that instinct is also true: 'we want to put as little as possible in the OS kernel, because that stuff can bring down the machine.'
That isn't true, as there are many things that can bring down the machine. If your X or Wayland server crashes, it brings down your machine.
No, that is true, even if not uniquely. A kernel bug can easily bring down the machine. Maybe Xorg can also manage it... although IMHO that also sounds like a kernel bug (surely it's the kernel's job to ensure no user program can do that?)... but that doesn't make the kernel safe. And I suspect it's easier to take down the machine from ring 0 than a display server even if both can do it.