Yocto can be incredibly simple, this is my favorite example: https://github.com/bootlin/simplest-yocto-setup/ Only the kernel and bootloader usually need to be specialized for most modern arm boards: the userland can be generic. Most of the problems people have with yocto are due to layers from hardware vendors which contain a lot of unnecessary cruft.
Yocto can appear incredibly simple. Until something somewhere deep inside the build process breaks, or you need to enable a peripheral the default device-tree for your board doesn't enable, or a gnat farts on the other side of the world, and it completely stops working.
If you buy hardware from a vendor who hands you a "meta-bigco" layer with their own fork of u-boot and the kernel, you're gonna have a bad time...