I don't think that's the reason. The reason is that they are unlikely to hurt anyone besides the pilot in a crash.
They can have an engine failure, stall-spin, or fly into a cloud just as readily as any other plane. Those are the biggest risks for GA, and they're no easier to handle in an ultralight.