Definitely not "key". On the ground fatalities are very rare, perhaps 1%-2% of all general aviation fatalities [1].
One massive restriction on ultralight airplanes is that you can't carry passengers. Again, on paper, because of risk, but it ends up having some controversial effects. If you want to occasionally carry passengers, as many pilots do, you can't fly a cheap, simple, light, and slow ultralight aircraft. No, the regulations push you to fly a more expensive, complex, heavier, and faster one that is type-certified, but requires higher skill and currency (regularly recurring flight experience) to operate safely.
Given that pilot error is far and away the main cause of general aviation crashes, that push does not seem very wise, safety-wise, even disregarding the question of costs.
> A few thousand pounds moving at up to 200mph or so (potentially 300+ after structural failure) is a lot of kinetic energy.
Also, FTR, Cessna 152, a type-certified two seater, weights 1670 lbs gross, cruises at 120mph, and if it ever hits the ground wrong, it very likely won't be because it broke up in mid-air and plummeted at terminal velocity onto someone's house, but because the pilot stalled it at low altitude while turning from base to final, crashing onto airport property or an empty field nearby.
[1] See table 10 in the accident statistics spreadsheet here: https://www.ntsb.gov/safety/Pages/research.aspx