Proximity flying is just inherently dangerous. Even with power, many people have died. Sparky Imeson, who wrote the "bible" of mountain flying, died from flying too close to terrain. The strange thing is that he was actually flying back to somewhere that he had had an accident in the past (the first time he just got seriously injured, the second time he died). If you look at the accident location, there is a valley at right-angles to the valley they're flying up, possibly causing katabatic downdrafts or something similar.
Steve Fossett apparently died flying too close to terrain at high density altitude with forecast winds possibly resulting in downdrafts.
Both of those people were extremely experienced pilots, but they just cut the margin for error too low. The main problem seems to be that winds are somewhat unpredictable, and if you're flying too close to terrain you might not have a chance to recover if you hit a sudden downdraft. I can't imagine doing this with a powered aircraft, never mind a wingsuit.
These pilots who died were 100-300ft from terrain, never mind a few feet like these unpowered wingsuit pilots. If you stall from 100ft you're dead in almost any type of aircraft as there isn't enough altitude to recover. (I don't know how many of these wingsuit deaths were caused by stalls. It seems equally likely that they just misjudged things or a small change in wind just pushed them into terrain).
There's gotta be safer ways to get an adrenaline rush.