If the speed is the same, and the places traveled are the same, and the likelihood of an accident is the same...
Can you imagine going to a car dealership, and coming across a car with 10x the stopping distance and a center of mass twice as high as a Civic and the salesman telling you "it's just as safe so long as you're traveling the same speed and same places"? Would you buy that claim for a second?
I've avoided numerous accidents by being able to stop quickly on my bike, or by making a sharp turn. This varies greatly just between different bicycles -- a mountain bike has much better stopping distance than a road bike. There's obviously also a big difference between a skateboard and a bicycle.
There's also, I think, a steeper learning curve between skateboard and bike, though that may be because learning to bike as a child is more common. There's at least a difference in skill levels. I mostly know the limits of my bike. (e.g., how hard can I turn without skidding out?) and while a skilled skateboarder may be on par with me on that, I'm not. I can bike tens of thousands of miles without falling. I'm nowhere near the skill level of being able to do that on a skateboard.
Being able to eject yourself is also valuable. I suspect a skateboarder would have better "eject and run" options than a cyclist, so that's a point in their favor. (I've had my back tire run over twice, and both times I was able to simply run forward over the handlebars, and my hands (and head) didn't touch the grounds, but it's not always going to be that easy.)